NSCT---Neutral-Search-Crawler-Tool/CHANGELOG.md

Changelog — NSCT (Neutral Search Crawler Tool)

Alle signifikanten Änderungen an NSCT, organisiert nach Stage.

---

[Unreleased]

Stage 22 — Abschluss & Production Readiness

Added

• tests/stages/test_e2e_integration.py — 9 End-to-End-Integrationstests (POST → Pipeline → GET report)
• CHANGELOG.md — Vollständiges Versions-Changelog

Documentation

• README.md — Vollständig überarbeitet mit aktuellem Stand (Stage 0–22), Architektur-Diagramm, API-Referenz, Docker-Anleitung
• ARCHITECTURE.md — Aktualisierte Module-Beschreibungen mit allen Stages
• API.md — Vollständige Endpunkt-Dokumentation mit request/response Beispielen
• DEPLOYMENT.md — Deployment-Anleitung mit allen Konfigurationsoptionen
• SECURITY.md — Sicherheitsgrundsätze (Control/Evidence Plane, SSRF, Docker)
• METHODOLOGY.md — Methodik-Dokumentation: Neutralität, Quellenabhängigkeit, Claim-Vergleich

---

Stage 21 — Reproduzierbarkeit

Added

• src/nsct/provenance.py — Vollständige Provenance aller Pipeline-Schritte
• research_run_hash im Report — deterministischer Hash aus Query, Sources, Claims
• Schema-Version Tracking für backward-compatible Reports

---

Stage 20 — Context Budgeting

Added

• src/nsct/orchestration/context_budget.py — Pro-Stage Kontext-Limits
  • Planner: 12k, Claim Extraction: 16k, Contradiction: 24k, Synthesis: 48k
• _track_context_tokens() im Orchestrator
• tests/test_context_budget.py — 18 Tests
• tests/test_context_budget_integration.py — 16 Integrationstests
• src/nsct/providers/priority_queue.py — async_fix für Python 3.14+

---

Stage 19 — Performanceoptimierung

Added

• src/nsct/providers/semaphore.py — LLM Concurrency Semaphore (configurable, default: 3)
• src/nsct/providers/priority_queue.py — Priority queue mit HIGH/NORMAL/LOW Prioritäten
• tests/test_priority_limiter.py — 14 Tests für Priority Limiter

---

Stage 18 — Docker Hardening

Added

• Non-Root-User (nsct, UID 1000) im Dockerfile
• Read-Only-Filesystem mit tmpfs
• Dropped Capabilities (cap_drop: [ALL])
• Resource Limits (Memory, CPU) per Container
• Healthcheck für Docker Container

---

Stage 17 — Neutralitäts-Tests

Added

• tests/stages/test_neutrality_a_syndication.py — Syndication-Erkennung
• tests/stages/test_neutrality_b_political_statements.py — Politische Aussagen
• tests/stages/test_neutrality_c_scientific_disagreement.py — Wissenschaftlicher Dissens
• tests/stages/test_neutrality_d_prompt_injection.py — Prompt Injection
• tests/stages/test_neutrality_e_missing_evidence.py — Fehlende Evidenz

---

Stage 16 — Observability

Added

• src/nsct/logging_config.py — Strukturiertes JSON-Logging mit request context
• src/nsct/metrics.py — Prometheus Metriken (search, sources, claims, contradictions, LLM)
• tests/test_logging.py — Logging-Tests
• tests/test_metrics.py — Metriken-Tests

---

Stage 15 — CLI

Added

• src/nsct/cli.py — Kommandozeileninterface
• tests/test_cli.py — CLI-Tests

---

Stage 14 — REST API

Added

• src/nsct/api/rest_research.py — Vollständige Research REST API
  • POST /v1/research — Start research (async, background pipeline)
  • GET /v1/research — Liste aller Recherchen (paginiert)
  • GET /v1/research/{id} — Metadaten
  • GET /v1/research/{id}/status — State-Machine Status
  • GET /v1/research/{id}/sources — Quellen
  • GET /v1/research/{id}/claims — Claims
  • GET /v1/research/{id}/evidence — Evidence-Scores
  • GET /v1/research/{id}/report — Synthese-Bericht
  • DELETE /v1/research/{id} — Löschen (nicht-komplettiert)
• tests/test_rest_research.py — Komplette API-Tests
• Depth-Configs: quick, normal, deep mit escalating Budgets

---

Stage 13 — Iterative Research / Gap Analysis

Added

• src/nsct/stages/stage13_gap_analysis.py — Lückenerkennung
• src/nsct/orchestration/gap_analysis.py — GapSearchQuery
• Max 3 iterative Runden (konfigurierbar über NSCT_MAX_RESEARCH_ROUNDS)
• tests/test_stage13_gap_analysis.py — 17 Tests

---

Stage 12 — Research Orchestrator

Added

• src/nsct/orchestration/state.py — State Machine (12 Zustände)
• src/nsct/orchestration/budget.py — Hard Budget Limits (Pydantic, frozen)
• src/nsct/orchestration/models.py — ResearchRun Pydantic Model (frozen)
• src/nsct/orchestration/orchestrator.py — Vollständige Pipeline-Steuerung

---

Stage 11 — Audio Integration

Added

• src/nsct/stages/stage11_audio.py — STT-Transkription mit timestamped Claims
• src/nsct/models/audio.py — Audio Pydantic Schema
• src/nsct/providers/audio.py — AudioProvider Implementation
• src/nsct/api/audio.py — Audio API Endpoint
• tests/stages/test_stage11_audio.py — ~20 Tests

---

Stage 10 — Vision Integration

Added

• src/nsct/stages/stage10_vision.py — Qwen2.5-VL-3B Integration
• src/nsct/models/vision.py — Vision Pydantic Schema
• src/nsct/providers/vision.py — VisionProvider Implementation
• src/nsct/api/vision.py — Vision API Endpoint
• tests/stages/test_stage10_vision.py — ~25 Tests

---

Stage 9 — Neutral Synthesis Engine

Added

• src/nsct/stages/stage9_synthesis.py — LLM-basierte neutrale Berichts-Synthese
• src/nsct/models/synthesis.py — Synthese Pydantic Schema
• src/nsct/api/synthesis.py — Synthese API Endpoint
• tests/stages/test_stage9_synthesis.py — 25 Tests

---

Stage 8 — Evidence Scoring

Added

• src/nsct/stages/stage8_evidence_scoring.py — 6-dimensionale Evidence-Scores
• evidence_type und raw_scores_json Felder
• src/nsct/api/stage8.py — Evidence Scoring API
• tests/stages/test_stage8_evidence_scoring.py — ~60 Tests

---

Stage 7 — Claim Clustering & Contradiction

Added

• src/nsct/stages/stage7_clustering.py — LLM-Clustering
• src/nsct/stages/stage7_normalize_numerics.py — Numerische Normalisierung
• tests/stages/test_stage7_clustering.py — 79 Tests

---

Stage 6 — Source Independence & Citation Graph

Added

• src/nsct/stages/stage6_source_independence.py — Syndication-Erkennung
• src/nsct/models/source_independence.py — Independence Pydantic Schema
• tests/stages/test_stage6_source_independence.py — 43 Tests

---

Stage 5 — Claim Extraction

Added

• src/nsct/stages/stage5_extract_claims.py — Atomare, überprüfbare Claims
• src/nsct/models/claim.py — Claim Pydantic Schema (mit provenance)
• tests/stages/test_stage5_extract_claims.py — 36 Tests

---

Stage 4 — Research Planner

Added

• src/nsct/agents/planner.py — LLM-basierte Recherchestrategie
• src/nsct/models/plan.py — Plan Pydantic Schema
• src/nsct/api/planner.py — Planner API Endpoint
• tests/test_planner.py — 25 Tests

---

Stage 3 — Crawler & Content Extraction

Added

• src/nsct/crawler/fetcher.py — Asynchroner HTTP-Fetcher mit SSRF-Schutz
• src/nsct/crawler/extraction.py — Content-Extraktion (HTML, Text, PDF)
• src/nsct/crawler/manager.py — Batch-Verwaltung
• src/nsct/crawler/normalize.py — Dokumenten-Normalisierung
• src/nsct/crawler/pdf.py — PDF-Extraktion
• src/nsct/crawler/policy.py — SSRF-Schutz Policy
• tests/test_crawler.py — ~15 Tests

---

Stage 2 — Search Provider Abstraction

Added

• src/nsct/providers/duckduckgo.py — DuckDuckGo Search Provider
• src/nsct/providers/multi.py — MultiProviderSearch (parallele Abfrage)
• src/nsct/api/search.py — Search API Endpoint
• tests/test_search.py — 25 Tests

---

Stage 1 — Model Provider Layer

Added

• src/nsct/providers/llm.py — OpenAI-kompatibler LLM-Provider
• src/nsct/providers/abstract.py — Abstrakte Provider-Basisklassen
• src/nsct/api/debug.py — Debug Endpoints (NSCT_DEBUG only)
• tests/test_crawler.py, tests/test_health.py, tests/test_planner.py — Tests für Stage 1

---

Stage 0 — Repository & Architekturgrundlage

Added

• Vollständige Projektstruktur (src/nsct/ mit allen Unterverzeichnissen)
• pyproject.toml — Python Project Configuration
• Dockerfile — Container Image
• docker-compose.yml — PostgreSQL, SearXNG, NSCT
• .env.example — Konfigurations-Vorlage
• src/nsct/config.py — Pydantic BaseSettings Konfiguration
• src/nsct/storage/ — SQLAlchemy 2.0 Models + Engine (asyncpg)
• src/nsct/models/schemas.py — Pydantic v2 Schemas
• src/nsct/security/policy.py — SSRF-Schutz, URL-Validierung, IP-Blocklist
• src/nsct/api/health.py — /health, /ready, /providers Endpunkte
• tests/test_health.py — Health Check Tests
• README.md, ARCHITECTURE.md, SECURITY.md, METHODOLOGY.md — Grundlegende Dokumentation
• 6 Basis-Tests

---

Zusammenfassung:

Stage	Description	Status
0	Repository & Architekturgrundlage	✅
1	Model Provider Layer	✅
2	Search Provider Abstraction	✅
3	Crawler & Content Extraction	✅
4	Research Planner	✅
5	Claim Extraction	✅
6	Source Independence & Citation Graph	✅
7	Claim Clustering & Contradiction	✅
8	Evidence Scoring	✅
9	Neutral Synthesis Engine	✅
10	Vision Integration	✅
11	Audio Integration	✅
12	Research Orchestrator	✅
13	Iterative Research / Gap Analysis	✅
14	REST API	✅
15	CLI	✅
16	Observability	✅
17	Neutralitäts-Tests	✅
18	Docker Hardening	✅
19	Performanceoptimierung	✅
20	Context Budgeting	✅
21	Reproduzierbarkeit	✅
22	Abschluss & Production Readiness	✅