opencode/NSCT---Neutral-Search-Crawler-Tool
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix research pipeline provider failures

Browse Source
This commit is contained in:
faligam
2026-09-06 18:32:42 +02:00
parent 9f6029b60a
commit e5b7d90436
9 changed files with 269 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
HANDOFF.md
View File

@@ -85,39 +85,34 @@ Mit einem gültigen, bereits erzeugten Key im Browser testen:
Antwortcodes gemeinsam auswerten. Klartext-Keys niemals in Chat oder Logs
kopieren.
### Aktueller Blocker im Browser-Flow — 2026-09-06
### Browser-Flow: Backend-Blocker lokal behoben — 2026-09-06
Der Browser-Flow erreicht inzwischen die Detailseite und fragt Status,
Quellen, Claims, Evidenz und Bericht ab. Eine tatsächlich gestartete Recherche
liefert jedoch derzeit keine Ergebnisse. Dies ist ein Backend-Problem, nicht
mehr der Frontend-Proxy oder die API-Key-Loginstrecke.
Die drei Backend-Ursachen für leere, scheinbar erfolgreiche Researches sind
lokal implementiert, getestet und mit `docker compose up -d --build` deployed:
Reproduzierbare Logbefunde aus `docker compose logs nsct-api`:
1. `LLMProvider._create_client()` normalisiert die Basis-URL und hängt `/v1`
nur an, wenn sie nicht bereits enthalten ist. Ein authentifizierter,
sanitisiert ausgegebener Chat-Completions-Probe war erfolgreich.
2. `SearXNGProvider` ist in `src/nsct/providers/searxng.py` implementiert und
wird vom Orchestrator registriert. Compose mountet eine minimale SearXNG-
Konfiguration, die `format=json` erlaubt, und nutzt den vom aktuellen
SearXNG-Image erwarteten Variablennamen `SEARXNG_SECRET`. Ein echter,
sanitisiert ausgegebener Search-Probe lieferte ein Ergebnis.
3. Die Pipeline führt wieder `fetching → extracting → analyzing` aus.
Fehler beim Planen, Suchen, Abrufen oder Extrahieren werden nicht mehr als
leere Fallbacks als Erfolg gemeldet: Der Run wird `FAILED`, mit Fehlergrund
im REST-Status (`error`). Fehlende Provider führen ebenfalls zu `FAILED`.
- `Planning failed: Error code: 401 - {'error': 'authentication required'}`
- `No search providers configured — search will return empty results`
- abgelehnte State-Transitions, etwa `fetching -> analyzing`,
`fetching -> synthesizing` und `fetching -> completed`.
Validierung: Syntaxprüfung, Compose-Konfigurationsprüfung und 29 fokussierte
Tests (inklusive neuer Regressionen für URL-Normalisierung, SearXNG und
State-Flow) sind erfolgreich. `tests/test_rest_research.py` wurde nicht als
Gesamtsuite abgewartet, da seine Background-Tasks echte externe Retries
auslösen; die neuen deterministischen Tests decken den geänderten Fehlerpfad.
Ursachen und nächste Reparaturen:
1. `src/nsct/providers/llm.py` ergänzt in `_create_client()` immer `/v1`.
`NSCT_LLM_BASE_URL` ist laut `.env.example` und Deployment-Dokumentation
bereits eine OpenAI-kompatible URL mit `/v1`. Den Basis-URL-Join
normalisieren, damit nie `/v1/v1` entsteht. Danach einen authentifizierten
Chat-Completions-Probe ausführen, ohne URL oder Key auszugeben.
2. `src/nsct/orchestration/orchestrator.py::_get_multi_search()` importiert
`nsct.providers.searxng.SearXNGProvider`, aber
`src/nsct/providers/searxng.py` existiert nicht. Der `ImportError` wird
still geschluckt; deshalb hat die Pipeline keinen Search-Provider. Entweder
den SearXNG-Provider implementieren oder den vorhandenen
`DuckDuckGoProvider` als Fallback registrieren. Compose setzt bereits
`NSCT_SEARXNG_BASE_URL=http://searxng:8080/`.
3. Das Orchestrator-/REST-Fehlerhandling muss bei nicht ausführbarer Pipeline
einen nachvollziehbaren `FAILED`-Status mitsamt Fehlergrund liefern, statt
mit leeren Fallbacks scheinbar abgeschlossene Schritte zu zeigen. Die
Transition-Matrix und die Fallback-Logik in `orchestrator.py` prüfen und
gezielt testen.
Nächster Schritt: Den Browser-End-to-End-Test mit einem gültigen, **nicht im
Chat offengelegten** Frontend-API-Key durchführen. Dabei insbesondere prüfen,
dass Quellen, Claims und Bericht nach einem echten Research-Lauf erscheinen
und ein Provider-/LLM-Ausfall in der UI den REST-Fehlergrund zeigt.
Frontend-Fixes des laufenden Browser-Tests (alle im Frontend-Repository
`/home/faligam/apps/NSCT-FrontEnd`, `main`, gepusht):

8
config/searxng/settings.yml Normal file
View File

@@ -0,0 +1,8 @@
# NSCT needs SearXNG's structured result endpoint. Keep HTML enabled for
# manual administration while allowing the backend's `format=json` requests.
use_default_settings: true
search:
formats:
- html
- json

4
docker-compose.yml
View File

@@ -52,9 +52,10 @@ services:
- "8888:8080"
volumes:
- searxng_cache:/var/cache/searxng
- ./config/searxng/settings.yml:/etc/searxng/settings.yml:ro
environment:
- SEARXNG_BASE_URL=http://localhost:8888/
- SEARXNG_SECRET_KEY=nsct_searxng_secret_key_change_me
- SEARXNG_SECRET=nsct_searxng_secret_key_change_me
restart: unless-stopped
deploy:
resources:
@@ -64,7 +65,6 @@ services:
read_only: true
tmpfs:
- /tmp
- /etc/searxng:mode=1777
cap_drop:
- ALL

5
src/nsct/api/rest_research.py
View File

@@ -147,6 +147,7 @@ class StatusResponse(BaseModel):
search_count: int = 0
source_count: int = 0
claim_count: int = 0
error: str | None = None
is_completed: bool = False
is_running: bool = False
@@ -468,6 +469,8 @@ async def _run_pipeline(run: _ResearchRunState, request: ResearchRequest, budget
)
else:
run.state = ResearchRunState.FAILED.value
run.metadata["error"] = result.get("error", "Pipeline failed")
run.metadata["failed_step"] = result.get("failed_step", "unknown")
metrics.increment(C_RESEARCH_FAILED_TOTAL)
logger.warning(
"[pipeline] run_id=%s failed: %s",
@@ -519,6 +522,7 @@ async def get_research(research_id: str) -> StatusResponse:
search_count=len(run.search_results),
source_count=len(run.sources),
claim_count=len(run.claims),
error=run.metadata.get("error"),
is_completed=(run.state == ResearchRunState.COMPLETED.value),
is_running=run.state not in (
ResearchRunState.COMPLETED.value,
@@ -546,6 +550,7 @@ async def get_status(research_id: str) -> StatusResponse:
search_count=len(run.search_results),
source_count=len(run.sources),
claim_count=len(run.claims),
error=run.metadata.get("error"),
is_completed=(run.state == ResearchRunState.COMPLETED.value),
is_running=run.state not in (
ResearchRunState.COMPLETED.value,

63
src/nsct/orchestration/orchestrator.py
View File

@@ -208,8 +208,6 @@ class ResearchOrchestrator:
"""
logger.error("Research failed: %s (state=%s)", reason, self._state_machine.current_state.value)
# Der State Machine _apply erzwingt FAILED direkt,
# weil _mark_failed als "Sondertransition" gedacht ist.
if self._state_machine.current_state in (
ResearchRunState.COMPLETED,
ResearchRunState.FAILED,
@@ -217,8 +215,16 @@ class ResearchOrchestrator:
):
return False
self._state_machine.current_state = ResearchRunState.FAILED # type: ignore[assignment]
return True
transitioned = self._transition_to(ResearchRunState.FAILED.value)
if transitioned and self._run is not None:
self._run = self._run.model_copy(
update={
"state": ResearchRunState.FAILED.value,
"updated_at": datetime.now(timezone.utc),
"metadata": {**self._run.metadata, "error": reason},
}
)
return transitioned
# ---------------------------------------------------------------
# Private: Budget
@@ -297,14 +303,14 @@ class ResearchOrchestrator:
if self._config.searxng_base_url:
provider = SearXNGProvider(base_url=self._config.searxng_base_url)
providers.append(provider)
except ImportError:
pass
except ImportError as exc:
logger.error("SearXNG provider is unavailable: %s", exc)
except Exception as exc:
logger.warning("SearXNG provider init failed (%s) — search will return empty", exc)
logger.error("SearXNG provider initialization failed: %s", exc)
self._multi_search = MultiProviderSearch(providers=providers if providers else None)
self._multi_search = MultiProviderSearch(providers=providers)
if not providers:
logger.warning("No search providers configured — search will return empty results")
logger.error("No search providers configured")
return self._multi_search
def _get_llm_provider(self):
@@ -406,6 +412,7 @@ class ResearchOrchestrator:
"planning",
"searching",
"fetching",
"extracting",
"analyzing",
"comparing",
"synthesizing",
@@ -440,6 +447,7 @@ class ResearchOrchestrator:
"success": False,
"error": error_msg,
"failed_step": step_name,
"state": self._state_machine.current_state.value,
"report": None,
}
@@ -458,6 +466,7 @@ class ResearchOrchestrator:
"success": False,
"error": str(exc),
"failed_step": step_name,
"state": self._state_machine.current_state.value,
"report": None,
}
@@ -552,18 +561,7 @@ class ResearchOrchestrator:
except Exception as exc:
logger.error("Planning failed: %s", exc)
# Fallback: Create a minimal plan
fallback = self._get_fallback_plan()
self._plan = fallback
# Store error in _run metadata via new instance
if self._run is not None:
self._run = self._run.model_copy(
update={
"plan_error": str(exc),
"updated_at": datetime.utcnow(),
}
)
return {"success": True, "data": {"plan": fallback}, "plan": fallback, "warning": "Used fallback plan"}
return {"success": False, "error": f"Planning failed: {exc}", "data": {}}
async def _step_searching(self) -> dict[str, Any]:
"""SEARCHING: Führe Suchanfragen durch und sammle URLs."""
@@ -571,6 +569,12 @@ class ResearchOrchestrator:
try:
multi_search = self._get_multi_search()
if not multi_search.enabled_providers():
return {
"success": False,
"error": "No search provider is configured or available",
"data": {},
}
plan_queries = self._plan.get("queries", []) if self._plan else []
if not plan_queries:
@@ -615,9 +619,8 @@ class ResearchOrchestrator:
return {"success": True, "data": {"urls": self._search_results}, "url_count": len(self._search_results)}
except Exception as exc:
logger.warning("Searching failed (graceful fallback): %s — returning empty results", exc)
self._search_results = []
return {"success": True, "data": {"urls": []}, "url_count": 0, "warning": str(exc)}
logger.error("Searching failed: %s", exc)
return {"success": False, "error": f"Searching failed: {exc}", "data": {}}
async def _step_fetching(self) -> dict[str, Any]:
"""FETCHING: Crawle gesammelte URLs und extrahiere Inhalt."""
@@ -669,9 +672,8 @@ class ResearchOrchestrator:
}
except Exception as exc:
logger.warning("Fetching failed (graceful fallback): %s — returning empty sources", exc)
self._sources = []
return {"success": True, "data": {"sources": []}, "source_count": 0, "warning": str(exc)}
logger.error("Fetching failed: %s", exc)
return {"success": False, "error": f"Fetching failed: {exc}", "data": {}}
async def _step_extracting(self) -> dict[str, Any]:
"""EXTRACTING: Extrahiere Claims aus den extrahierten Quellen."""
@@ -717,9 +719,8 @@ class ResearchOrchestrator:
return {"success": True, "data": {"claims": [c.model_dump() for c in self._claims]}, "claim_count": len(self._claims)}
except Exception as exc:
logger.warning("Claim extraction failed (graceful fallback): %s", exc)
self._claims = []
return {"success": True, "data": {"claims": []}, "claim_count": 0, "warning": str(exc)}
logger.error("Claim extraction failed: %s", exc)
return {"success": False, "error": f"Claim extraction failed: {exc}", "data": {}}
async def _step_analyzing(self) -> dict[str, Any]:
"""ANALYZING: Platzhalter — Stage 6/7/8 werden später eingebunden.
@@ -1180,4 +1181,4 @@ class ResearchOrchestrator:
except Exception as exc:
logger.warning("Gap search execution failed: %s", exc)
return {"search_results": search_results, "claims": claims}
return {"search_results": search_results, "claims": claims}

2
src/nsct/orchestration/state.py
View File

@@ -39,7 +39,7 @@ class ResearchRunState(str, Enum):
# Übergangsmatrix: Quelle -> zulässige Ziele (frozenset für Immutable)
_VALID_TRANSITIONS: dict[ResearchRunState, FrozenSet[ResearchRunState]] = {
ResearchRunState.CREATED: frozenset(
(ResearchRunState.PLANNING,)
(ResearchRunState.PLANNING, ResearchRunState.FAILED)
),
ResearchRunState.PLANNING: frozenset(
(ResearchRunState.SEARCHING, ResearchRunState.FAILED)

10
src/nsct/providers/llm.py
View File

@@ -151,8 +151,14 @@ class _LLMProviderImpl(LLMProvider):
pool=5,
),
)
# OpenAI-compatible endpoints are commonly configured either as the
# service root or with the API version already included. Appending
# blindly turns an explicit ``.../v1`` into ``.../v1/v1``.
base_url = self._base_url.rstrip("/")
if not base_url.endswith("/v1"):
base_url += "/v1"
return AsyncOpenAI(
base_url=self._base_url.rstrip("/") + "/v1",
base_url=base_url,
api_key=self._api_key,
http_client=http_client,
)
@@ -354,4 +360,4 @@ def get_provider(config: AppSettings, metrics: ProviderMetrics) -> LLMProvider:
global _instance
if _instance is None:
_instance = _LLMProviderImpl(config, metrics)
return _instance
return _instance

82
src/nsct/providers/searxng.py Normal file
View File

@@ -0,0 +1,82 @@
"""SearXNG JSON search provider."""
from __future__ import annotations
from typing import Any
import httpx
from nsct.providers.abstract import NormalizedResult, SearchProvider
class SearXNGProvider(SearchProvider):
"""Search through a self-hosted SearXNG instance's JSON endpoint."""
_provider_name = "searxng"
def __init__(self, base_url: str, timeout_seconds: float = 15.0) -> None:
if not base_url.strip():
raise ValueError("SearXNG base URL must not be empty")
self._base_url = base_url.rstrip("/")
self._timeout_seconds = timeout_seconds
self._client: httpx.AsyncClient | None = None
@property
def _http_client(self) -> httpx.AsyncClient:
if self._client is None:
self._client = httpx.AsyncClient(timeout=httpx.Timeout(self._timeout_seconds))
return self._client
async def search(
self, query: str, language: str = "de", max_results: int = 10
) -> list[NormalizedResult]:
response = await self._http_client.get(
f"{self._base_url}/search",
params={"q": query, "format": "json", "language": language},
# The bundled SearXNG instance enables bot detection and rejects
# requests without a client IP header. This is the loopback IP of
# the API-to-SearXNG hop, not an end-user supplied value.
headers={"X-Forwarded-For": "127.0.0.1"},
)
response.raise_for_status()
payload = response.json()
raw_results = payload.get("results", [])
if not isinstance(raw_results, list):
raise ValueError("Invalid SearXNG response: results is not a list")
results: list[NormalizedResult] = []
for rank, item in enumerate(raw_results[:max_results], start=1):
if not isinstance(item, dict) or not isinstance(item.get("url"), str):
continue
url = item["url"]
if not url.startswith(("http://", "https://")):
continue
results.append(
NormalizedResult.from_raw(
provider_name=self._provider_name,
title=str(item.get("title") or url),
url=url,
snippet=str(item.get("content") or item.get("snippet") or ""),
rank=rank,
extra={"engine": item.get("engine"), "category": item.get("category")},
)
)
return results
async def get_metadata(self) -> dict[str, Any]:
return {
"name": self._provider_name,
"version": "0.1.0",
"provider": self.__class__.__name__,
"capabilities": ["web_search", "json_api"],
"requires_api_key": False,
"base_url": self._base_url,
"note": "Search ranking is NOT a trust indicator.",
}
async def health_check(self) -> bool:
try:
response = await self._http_client.get(f"{self._base_url}/healthz")
return response.is_success
except httpx.HTTPError:
return False

107
tests/test_backend_pipeline_repairs.py Normal file
View File

@@ -0,0 +1,107 @@
"""Regression tests for the deployed backend research pipeline fixes."""
from __future__ import annotations
import asyncio
from uuid import uuid4
import httpx
from unittest.mock import AsyncMock
from nsct.config import AppSettings, AudioConfig, DatabaseConfig, LLMConfig, VisionConfig
from nsct.orchestration.orchestrator import ResearchOrchestrator
from nsct.orchestration.state import ResearchRunState
from nsct.providers.llm import _LLMProviderImpl
from nsct.providers.metrics import ProviderMetrics
from nsct.providers.searxng import SearXNGProvider
def _config(*, searxng_base_url: str | None = "http://searxng:8080") -> AppSettings:
return AppSettings(
llm=LLMConfig(base_url="http://llm.example/v1", model="test-model"),
vision=VisionConfig(base_url="http://vision.example/v1", model="vision"),
audio=AudioConfig(base_url="http://audio.example/v1"),
postgres=DatabaseConfig(url="sqlite+aiosqlite://"),
searxng_base_url=searxng_base_url,
)
def test_llm_base_url_does_not_duplicate_existing_v1() -> None:
provider = _LLMProviderImpl(_config(), ProviderMetrics())
provider._api_key = "test-key"
client = provider._create_client()
try:
assert str(client.base_url) == "http://llm.example/v1/"
finally:
asyncio.run(client.close())
def test_searxng_provider_normalizes_json_results() -> None:
async def run() -> None:
provider = SearXNGProvider("http://searxng:8080")
provider._client = httpx.AsyncClient(
transport=httpx.MockTransport(
lambda request: httpx.Response(
200,
json={
"results": [
{"url": "https://example.org/a", "title": "A", "content": "Snippet"},
{"url": "mailto:invalid@example.org", "title": "Invalid"},
]
},
)
)
)
try:
results = await provider.search("test")
assert len(results) == 1
assert results[0].provider == "searxng"
assert results[0].url == "https://example.org/a"
assert results[0].snippet == "Snippet"
assert provider._client is not None
finally:
await provider._client.aclose()
asyncio.run(run())
def test_orchestrator_registers_searxng_provider() -> None:
orchestrator = ResearchOrchestrator(_config(), uuid4(), "test")
provider = orchestrator._get_multi_search().get_provider("searxng")
assert isinstance(provider, SearXNGProvider)
def test_pipeline_executes_extracting_between_fetching_and_analyzing() -> None:
async def run() -> None:
orchestrator = ResearchOrchestrator(_config(), uuid4(), "test")
orchestrator._comparison_data = {}
for step in ("planning", "searching", "fetching", "extracting", "analyzing", "comparing", "synthesizing"):
async def handler(target: str = step) -> dict[str, object]:
orchestrator._transition_to(target)
return {"success": True, "data": {}}
setattr(orchestrator, f"_step_{step}", AsyncMock(side_effect=handler))
result = await orchestrator.run()
assert result["success"] is True
assert orchestrator._step_extracting.await_count == 1
assert orchestrator.state is ResearchRunState.COMPLETED
asyncio.run(run())
def test_no_search_provider_marks_run_failed_with_reason() -> None:
async def run() -> None:
orchestrator = ResearchOrchestrator(_config(searxng_base_url=None), uuid4(), "test")
await orchestrator.start()
orchestrator._step_planning = AsyncMock(return_value={"success": True, "data": {}})
result = await orchestrator.run()
assert result["success"] is False
assert result["failed_step"] == "searching"
assert "No search provider" in result["error"]
assert orchestrator.state is ResearchRunState.FAILED
asyncio.run(run())