stage22: Abschluss & Production Readiness — E2E-Tests, CHANGELOG, Dokumentation

CHANGELOG.md

@@ -0,0 +1,290 @@
+# Changelog — NSCT (Neutral Search Crawler Tool)
+
+Alle signifikanten Änderungen an NSCT, organisiert nach Stage.
+
+---
+
+## [Unreleased]
+
+### Stage 22 — Abschluss & Production Readiness
+
+**Added**
+- `tests/stages/test_e2e_integration.py` — 9 End-to-End-Integrationstests (POST → Pipeline → GET report)
+- `CHANGELOG.md` — Vollständiges Versions-Changelog
+
+**Documentation**
+- `README.md` — Vollständig überarbeitet mit aktuellem Stand (Stage 0–22), Architektur-Diagramm, API-Referenz, Docker-Anleitung
+- `ARCHITECTURE.md` — Aktualisierte Module-Beschreibungen mit allen Stages
+- `API.md` — Vollständige Endpunkt-Dokumentation mit request/response Beispielen
+- `DEPLOYMENT.md` — Deployment-Anleitung mit allen Konfigurationsoptionen
+- `SECURITY.md` — Sicherheitsgrundsätze (Control/Evidence Plane, SSRF, Docker)
+- `METHODOLOGY.md` — Methodik-Dokumentation: Neutralität, Quellenabhängigkeit, Claim-Vergleich
+
+---
+
+## Stage 21 — Reproduzierbarkeit
+
+**Added**
+- `src/nsct/provenance.py` — Vollständige Provenance aller Pipeline-Schritte
+- `research_run_hash` im Report — deterministischer Hash aus Query, Sources, Claims
+- Schema-Version Tracking für backward-compatible Reports
+
+---
+
+## Stage 20 — Context Budgeting
+
+**Added**
+- `src/nsct/orchestration/context_budget.py` — Pro-Stage Kontext-Limits
+  - Planner: 12k, Claim Extraction: 16k, Contradiction: 24k, Synthesis: 48k
+- `_track_context_tokens()` im Orchestrator
+- `tests/test_context_budget.py` — 18 Tests
+- `tests/test_context_budget_integration.py` — 16 Integrationstests
+- `src/nsct/providers/priority_queue.py` — async_fix für Python 3.14+
+
+---
+
+## Stage 19 — Performanceoptimierung
+
+**Added**
+- `src/nsct/providers/semaphore.py` — LLM Concurrency Semaphore (configurable, default: 3)
+- `src/nsct/providers/priority_queue.py` — Priority queue mit HIGH/NORMAL/LOW Prioritäten
+- `tests/test_priority_limiter.py` — 14 Tests für Priority Limiter
+
+---
+
+## Stage 18 — Docker Hardening
+
+**Added**
+- Non-Root-User (`nsct`, UID 1000) im Dockerfile
+- Read-Only-Filesystem mit tmpfs
+- Dropped Capabilities (`cap_drop: [ALL]`)
+- Resource Limits (Memory, CPU) per Container
+- Healthcheck für Docker Container
+
+---
+
+## Stage 17 — Neutralitäts-Tests
+
+**Added**
+- `tests/stages/test_neutrality_a_syndication.py` — Syndication-Erkennung
+- `tests/stages/test_neutrality_b_political_statements.py` — Politische Aussagen
+- `tests/stages/test_neutrality_c_scientific_disagreement.py` — Wissenschaftlicher Dissens
+- `tests/stages/test_neutrality_d_prompt_injection.py` — Prompt Injection
+- `tests/stages/test_neutrality_e_missing_evidence.py` — Fehlende Evidenz
+
+---
+
+## Stage 16 — Observability
+
+**Added**
+- `src/nsct/logging_config.py` — Strukturiertes JSON-Logging mit request context
+- `src/nsct/metrics.py` — Prometheus Metriken (search, sources, claims, contradictions, LLM)
+- `tests/test_logging.py` — Logging-Tests
+- `tests/test_metrics.py` — Metriken-Tests
+
+---
+
+## Stage 15 — CLI
+
+**Added**
+- `src/nsct/cli.py` — Kommandozeileninterface
+- `tests/test_cli.py` — CLI-Tests
+
+---
+
+## Stage 14 — REST API
+
+**Added**
+- `src/nsct/api/rest_research.py` — Vollständige Research REST API
+  - `POST /v1/research` — Start research (async, background pipeline)
+  - `GET /v1/research` — Liste aller Recherchen (paginiert)
+  - `GET /v1/research/{id}` — Metadaten
+  - `GET /v1/research/{id}/status` — State-Machine Status
+  - `GET /v1/research/{id}/sources` — Quellen
+  - `GET /v1/research/{id}/claims` — Claims
+  - `GET /v1/research/{id}/evidence` — Evidence-Scores
+  - `GET /v1/research/{id}/report` — Synthese-Bericht
+  - `DELETE /v1/research/{id}` — Löschen (nicht-komplettiert)
+- `tests/test_rest_research.py` — Komplette API-Tests
+- Depth-Configs: `quick`, `normal`, `deep` mit escalating Budgets
+
+---
+
+## Stage 13 — Iterative Research / Gap Analysis
+
+**Added**
+- `src/nsct/stages/stage13_gap_analysis.py` — Lückenerkennung
+- `src/nsct/orchestration/gap_analysis.py` — GapSearchQuery
+- Max 3 iterative Runden (konfigurierbar über `NSCT_MAX_RESEARCH_ROUNDS`)
+- `tests/test_stage13_gap_analysis.py` — 17 Tests
+
+---
+
+## Stage 12 — Research Orchestrator
+
+**Added**
+- `src/nsct/orchestration/state.py` — State Machine (12 Zustände)
+- `src/nsct/orchestration/budget.py` — Hard Budget Limits (Pydantic, frozen)
+- `src/nsct/orchestration/models.py` — ResearchRun Pydantic Model (frozen)
+- `src/nsct/orchestration/orchestrator.py` — Vollständige Pipeline-Steuerung
+
+---
+
+## Stage 11 — Audio Integration
+
+**Added**
+- `src/nsct/stages/stage11_audio.py` — STT-Transkription mit timestamped Claims
+- `src/nsct/models/audio.py` — Audio Pydantic Schema
+- `src/nsct/providers/audio.py` — AudioProvider Implementation
+- `src/nsct/api/audio.py` — Audio API Endpoint
+- `tests/stages/test_stage11_audio.py` — ~20 Tests
+
+---
+
+## Stage 10 — Vision Integration
+
+**Added**
+- `src/nsct/stages/stage10_vision.py` — Qwen2.5-VL-3B Integration
+- `src/nsct/models/vision.py` — Vision Pydantic Schema
+- `src/nsct/providers/vision.py` — VisionProvider Implementation
+- `src/nsct/api/vision.py` — Vision API Endpoint
+- `tests/stages/test_stage10_vision.py` — ~25 Tests
+
+---
+
+## Stage 9 — Neutral Synthesis Engine
+
+**Added**
+- `src/nsct/stages/stage9_synthesis.py` — LLM-basierte neutrale Berichts-Synthese
+- `src/nsct/models/synthesis.py` — Synthese Pydantic Schema
+- `src/nsct/api/synthesis.py` — Synthese API Endpoint
+- `tests/stages/test_stage9_synthesis.py` — 25 Tests
+
+---
+
+## Stage 8 — Evidence Scoring
+
+**Added**
+- `src/nsct/stages/stage8_evidence_scoring.py` — 6-dimensionale Evidence-Scores
+- `evidence_type` und `raw_scores_json` Felder
+- `src/nsct/api/stage8.py` — Evidence Scoring API
+- `tests/stages/test_stage8_evidence_scoring.py` — ~60 Tests
+
+---
+
+## Stage 7 — Claim Clustering & Contradiction
+
+**Added**
+- `src/nsct/stages/stage7_clustering.py` — LLM-Clustering
+- `src/nsct/stages/stage7_normalize_numerics.py` — Numerische Normalisierung
+- `tests/stages/test_stage7_clustering.py` — 79 Tests
+
+---
+
+## Stage 6 — Source Independence & Citation Graph
+
+**Added**
+- `src/nsct/stages/stage6_source_independence.py` — Syndication-Erkennung
+- `src/nsct/models/source_independence.py` — Independence Pydantic Schema
+- `tests/stages/test_stage6_source_independence.py` — 43 Tests
+
+---
+
+## Stage 5 — Claim Extraction
+
+**Added**
+- `src/nsct/stages/stage5_extract_claims.py` — Atomare, überprüfbare Claims
+- `src/nsct/models/claim.py` — Claim Pydantic Schema (mit provenance)
+- `tests/stages/test_stage5_extract_claims.py` — 36 Tests
+
+---
+
+## Stage 4 — Research Planner
+
+**Added**
+- `src/nsct/agents/planner.py` — LLM-basierte Recherchestrategie
+- `src/nsct/models/plan.py` — Plan Pydantic Schema
+- `src/nsct/api/planner.py` — Planner API Endpoint
+- `tests/test_planner.py` — 25 Tests
+
+---
+
+## Stage 3 — Crawler & Content Extraction
+
+**Added**
+- `src/nsct/crawler/fetcher.py` — Asynchroner HTTP-Fetcher mit SSRF-Schutz
+- `src/nsct/crawler/extraction.py` — Content-Extraktion (HTML, Text, PDF)
+- `src/nsct/crawler/manager.py` — Batch-Verwaltung
+- `src/nsct/crawler/normalize.py` — Dokumenten-Normalisierung
+- `src/nsct/crawler/pdf.py` — PDF-Extraktion
+- `src/nsct/crawler/policy.py` — SSRF-Schutz Policy
+- `tests/test_crawler.py` — ~15 Tests
+
+---
+
+## Stage 2 — Search Provider Abstraction
+
+**Added**
+- `src/nsct/providers/duckduckgo.py` — DuckDuckGo Search Provider
+- `src/nsct/providers/multi.py` — MultiProviderSearch (parallele Abfrage)
+- `src/nsct/api/search.py` — Search API Endpoint
+- `tests/test_search.py` — 25 Tests
+
+---
+
+## Stage 1 — Model Provider Layer
+
+**Added**
+- `src/nsct/providers/llm.py` — OpenAI-kompatibler LLM-Provider
+- `src/nsct/providers/abstract.py` — Abstrakte Provider-Basisklassen
+- `src/nsct/api/debug.py` — Debug Endpoints (NSCT_DEBUG only)
+- `tests/test_crawler.py`, `tests/test_health.py`, `tests/test_planner.py` — Tests für Stage 1
+
+---
+
+## Stage 0 — Repository & Architekturgrundlage
+
+**Added**
+- Vollständige Projektstruktur (`src/nsct/` mit allen Unterverzeichnissen)
+- `pyproject.toml` — Python Project Configuration
+- `Dockerfile` — Container Image
+- `docker-compose.yml` — PostgreSQL, SearXNG, NSCT
+- `.env.example` — Konfigurations-Vorlage
+- `src/nsct/config.py` — Pydantic BaseSettings Konfiguration
+- `src/nsct/storage/` — SQLAlchemy 2.0 Models + Engine (asyncpg)
+- `src/nsct/models/schemas.py` — Pydantic v2 Schemas
+- `src/nsct/security/policy.py` — SSRF-Schutz, URL-Validierung, IP-Blocklist
+- `src/nsct/api/health.py` — `/health`, `/ready`, `/providers` Endpunkte
+- `tests/test_health.py` — Health Check Tests
+- `README.md`, `ARCHITECTURE.md`, `SECURITY.md`, `METHODOLOGY.md` — Grundlegende Dokumentation
+- 6 Basis-Tests
+
+---
+
+**Zusammenfassung:**
+
+| Stage | Description | Status |
+|-------|-------------|--------|
+| 0 | Repository & Architekturgrundlage | ✅ |
+| 1 | Model Provider Layer | ✅ |
+| 2 | Search Provider Abstraction | ✅ |
+| 3 | Crawler & Content Extraction | ✅ |
+| 4 | Research Planner | ✅ |
+| 5 | Claim Extraction | ✅ |
+| 6 | Source Independence & Citation Graph | ✅ |
+| 7 | Claim Clustering & Contradiction | ✅ |
+| 8 | Evidence Scoring | ✅ |
+| 9 | Neutral Synthesis Engine | ✅ |
+| 10 | Vision Integration | ✅ |
+| 11 | Audio Integration | ✅ |
+| 12 | Research Orchestrator | ✅ |
+| 13 | Iterative Research / Gap Analysis | ✅ |
+| 14 | REST API | ✅ |
+| 15 | CLI | ✅ |
+| 16 | Observability | ✅ |
+| 17 | Neutralitäts-Tests | ✅ |
+| 18 | Docker Hardening | ✅ |
+| 19 | Performanceoptimierung | ✅ |
+| 20 | Context Budgeting | ✅ |
+| 21 | Reproduzierbarkeit | ✅ |
+| 22 | Abschluss & Production Readiness | ✅ |