# Changelog — NSCT (Neutral Search Crawler Tool)

Alle signifikanten Änderungen an NSCT, organisiert nach Stage.

---

## [Unreleased]

### Stage 22 — Abschluss & Production Readiness

**Added**
- `tests/stages/test_e2e_integration.py` — 9 End-to-End-Integrationstests (POST → Pipeline → GET report)
- `CHANGELOG.md` — Vollständiges Versions-Changelog

**Documentation**
- `README.md` — Vollständig überarbeitet mit aktuellem Stand (Stage 0–22), Architektur-Diagramm, API-Referenz, Docker-Anleitung
- `ARCHITECTURE.md` — Aktualisierte Module-Beschreibungen mit allen Stages
- `API.md` — Vollständige Endpunkt-Dokumentation mit request/response Beispielen
- `DEPLOYMENT.md` — Deployment-Anleitung mit allen Konfigurationsoptionen
- `SECURITY.md` — Sicherheitsgrundsätze (Control/Evidence Plane, SSRF, Docker)
- `METHODOLOGY.md` — Methodik-Dokumentation: Neutralität, Quellenabhängigkeit, Claim-Vergleich

---

## Stage 21 — Reproduzierbarkeit

**Added**
- `src/nsct/provenance.py` — Vollständige Provenance aller Pipeline-Schritte
- `research_run_hash` im Report — deterministischer Hash aus Query, Sources, Claims
- Schema-Version Tracking für backward-compatible Reports

---

## Stage 20 — Context Budgeting

**Added**
- `src/nsct/orchestration/context_budget.py` — Pro-Stage Kontext-Limits
  - Planner: 12k, Claim Extraction: 16k, Contradiction: 24k, Synthesis: 48k
- `_track_context_tokens()` im Orchestrator
- `tests/test_context_budget.py` — 18 Tests
- `tests/test_context_budget_integration.py` — 16 Integrationstests
- `src/nsct/providers/priority_queue.py` — async_fix für Python 3.14+

---

## Stage 19 — Performanceoptimierung

**Added**
- `src/nsct/providers/semaphore.py` — LLM Concurrency Semaphore (configurable, default: 3)
- `src/nsct/providers/priority_queue.py` — Priority queue mit HIGH/NORMAL/LOW Prioritäten
- `tests/test_priority_limiter.py` — 14 Tests für Priority Limiter

---

## Stage 18 — Docker Hardening

**Added**
- Non-Root-User (`nsct`, UID 1000) im Dockerfile
- Read-Only-Filesystem mit tmpfs
- Dropped Capabilities (`cap_drop: [ALL]`)
- Resource Limits (Memory, CPU) per Container
- Healthcheck für Docker Container

---

## Stage 17 — Neutralitäts-Tests

**Added**
- `tests/stages/test_neutrality_a_syndication.py` — Syndication-Erkennung
- `tests/stages/test_neutrality_b_political_statements.py` — Politische Aussagen
- `tests/stages/test_neutrality_c_scientific_disagreement.py` — Wissenschaftlicher Dissens
- `tests/stages/test_neutrality_d_prompt_injection.py` — Prompt Injection
- `tests/stages/test_neutrality_e_missing_evidence.py` — Fehlende Evidenz

---

## Stage 16 — Observability

**Added**
- `src/nsct/logging_config.py` — Strukturiertes JSON-Logging mit request context
- `src/nsct/metrics.py` — Prometheus Metriken (search, sources, claims, contradictions, LLM)
- `tests/test_logging.py` — Logging-Tests
- `tests/test_metrics.py` — Metriken-Tests

---

## Stage 15 — CLI

**Added**
- `src/nsct/cli.py` — Kommandozeileninterface
- `tests/test_cli.py` — CLI-Tests

---

## Stage 14 — REST API

**Added**
- `src/nsct/api/rest_research.py` — Vollständige Research REST API
  - `POST /v1/research` — Start research (async, background pipeline)
  - `GET /v1/research` — Liste aller Recherchen (paginiert)
  - `GET /v1/research/{id}` — Metadaten
  - `GET /v1/research/{id}/status` — State-Machine Status
  - `GET /v1/research/{id}/sources` — Quellen
  - `GET /v1/research/{id}/claims` — Claims
  - `GET /v1/research/{id}/evidence` — Evidence-Scores
  - `GET /v1/research/{id}/report` — Synthese-Bericht
  - `DELETE /v1/research/{id}` — Löschen (nicht-komplettiert)
- `tests/test_rest_research.py` — Komplette API-Tests
- Depth-Configs: `quick`, `normal`, `deep` mit escalating Budgets

---

## Stage 13 — Iterative Research / Gap Analysis

**Added**
- `src/nsct/stages/stage13_gap_analysis.py` — Lückenerkennung
- `src/nsct/orchestration/gap_analysis.py` — GapSearchQuery
- Max 3 iterative Runden (konfigurierbar über `NSCT_MAX_RESEARCH_ROUNDS`)
- `tests/test_stage13_gap_analysis.py` — 17 Tests

---

## Stage 12 — Research Orchestrator

**Added**
- `src/nsct/orchestration/state.py` — State Machine (12 Zustände)
- `src/nsct/orchestration/budget.py` — Hard Budget Limits (Pydantic, frozen)
- `src/nsct/orchestration/models.py` — ResearchRun Pydantic Model (frozen)
- `src/nsct/orchestration/orchestrator.py` — Vollständige Pipeline-Steuerung

---

## Stage 11 — Audio Integration

**Added**
- `src/nsct/stages/stage11_audio.py` — STT-Transkription mit timestamped Claims
- `src/nsct/models/audio.py` — Audio Pydantic Schema
- `src/nsct/providers/audio.py` — AudioProvider Implementation
- `src/nsct/api/audio.py` — Audio API Endpoint
- `tests/stages/test_stage11_audio.py` — ~20 Tests

---

## Stage 10 — Vision Integration

**Added**
- `src/nsct/stages/stage10_vision.py` — Qwen2.5-VL-3B Integration
- `src/nsct/models/vision.py` — Vision Pydantic Schema
- `src/nsct/providers/vision.py` — VisionProvider Implementation
- `src/nsct/api/vision.py` — Vision API Endpoint
- `tests/stages/test_stage10_vision.py` — ~25 Tests

---

## Stage 9 — Neutral Synthesis Engine

**Added**
- `src/nsct/stages/stage9_synthesis.py` — LLM-basierte neutrale Berichts-Synthese
- `src/nsct/models/synthesis.py` — Synthese Pydantic Schema
- `src/nsct/api/synthesis.py` — Synthese API Endpoint
- `tests/stages/test_stage9_synthesis.py` — 25 Tests

---

## Stage 8 — Evidence Scoring

**Added**
- `src/nsct/stages/stage8_evidence_scoring.py` — 6-dimensionale Evidence-Scores
- `evidence_type` und `raw_scores_json` Felder
- `src/nsct/api/stage8.py` — Evidence Scoring API
- `tests/stages/test_stage8_evidence_scoring.py` — ~60 Tests

---

## Stage 7 — Claim Clustering & Contradiction

**Added**
- `src/nsct/stages/stage7_clustering.py` — LLM-Clustering
- `src/nsct/stages/stage7_normalize_numerics.py` — Numerische Normalisierung
- `tests/stages/test_stage7_clustering.py` — 79 Tests

---

## Stage 6 — Source Independence & Citation Graph

**Added**
- `src/nsct/stages/stage6_source_independence.py` — Syndication-Erkennung
- `src/nsct/models/source_independence.py` — Independence Pydantic Schema
- `tests/stages/test_stage6_source_independence.py` — 43 Tests

---

## Stage 5 — Claim Extraction

**Added**
- `src/nsct/stages/stage5_extract_claims.py` — Atomare, überprüfbare Claims
- `src/nsct/models/claim.py` — Claim Pydantic Schema (mit provenance)
- `tests/stages/test_stage5_extract_claims.py` — 36 Tests

---

## Stage 4 — Research Planner

**Added**
- `src/nsct/agents/planner.py` — LLM-basierte Recherchestrategie
- `src/nsct/models/plan.py` — Plan Pydantic Schema
- `src/nsct/api/planner.py` — Planner API Endpoint
- `tests/test_planner.py` — 25 Tests

---

## Stage 3 — Crawler & Content Extraction

**Added**
- `src/nsct/crawler/fetcher.py` — Asynchroner HTTP-Fetcher mit SSRF-Schutz
- `src/nsct/crawler/extraction.py` — Content-Extraktion (HTML, Text, PDF)
- `src/nsct/crawler/manager.py` — Batch-Verwaltung
- `src/nsct/crawler/normalize.py` — Dokumenten-Normalisierung
- `src/nsct/crawler/pdf.py` — PDF-Extraktion
- `src/nsct/crawler/policy.py` — SSRF-Schutz Policy
- `tests/test_crawler.py` — ~15 Tests

---

## Stage 2 — Search Provider Abstraction

**Added**
- `src/nsct/providers/duckduckgo.py` — DuckDuckGo Search Provider
- `src/nsct/providers/multi.py` — MultiProviderSearch (parallele Abfrage)
- `src/nsct/api/search.py` — Search API Endpoint
- `tests/test_search.py` — 25 Tests

---

## Stage 1 — Model Provider Layer

**Added**
- `src/nsct/providers/llm.py` — OpenAI-kompatibler LLM-Provider
- `src/nsct/providers/abstract.py` — Abstrakte Provider-Basisklassen
- `src/nsct/api/debug.py` — Debug Endpoints (NSCT_DEBUG only)
- `tests/test_crawler.py`, `tests/test_health.py`, `tests/test_planner.py` — Tests für Stage 1

---

## Stage 0 — Repository & Architekturgrundlage

**Added**
- Vollständige Projektstruktur (`src/nsct/` mit allen Unterverzeichnissen)
- `pyproject.toml` — Python Project Configuration
- `Dockerfile` — Container Image
- `docker-compose.yml` — PostgreSQL, SearXNG, NSCT
- `.env.example` — Konfigurations-Vorlage
- `src/nsct/config.py` — Pydantic BaseSettings Konfiguration
- `src/nsct/storage/` — SQLAlchemy 2.0 Models + Engine (asyncpg)
- `src/nsct/models/schemas.py` — Pydantic v2 Schemas
- `src/nsct/security/policy.py` — SSRF-Schutz, URL-Validierung, IP-Blocklist
- `src/nsct/api/health.py` — `/health`, `/ready`, `/providers` Endpunkte
- `tests/test_health.py` — Health Check Tests
- `README.md`, `ARCHITECTURE.md`, `SECURITY.md`, `METHODOLOGY.md` — Grundlegende Dokumentation
- 6 Basis-Tests

---

**Zusammenfassung:**

| Stage | Description | Status |
|-------|-------------|--------|
| 0 | Repository & Architekturgrundlage | ✅ |
| 1 | Model Provider Layer | ✅ |
| 2 | Search Provider Abstraction | ✅ |
| 3 | Crawler & Content Extraction | ✅ |
| 4 | Research Planner | ✅ |
| 5 | Claim Extraction | ✅ |
| 6 | Source Independence & Citation Graph | ✅ |
| 7 | Claim Clustering & Contradiction | ✅ |
| 8 | Evidence Scoring | ✅ |
| 9 | Neutral Synthesis Engine | ✅ |
| 10 | Vision Integration | ✅ |
| 11 | Audio Integration | ✅ |
| 12 | Research Orchestrator | ✅ |
| 13 | Iterative Research / Gap Analysis | ✅ |
| 14 | REST API | ✅ |
| 15 | CLI | ✅ |
| 16 | Observability | ✅ |
| 17 | Neutralitäts-Tests | ✅ |
| 18 | Docker Hardening | ✅ |
| 19 | Performanceoptimierung | ✅ |
| 20 | Context Budgeting | ✅ |
| 21 | Reproduzierbarkeit | ✅ |
| 22 | Abschluss & Production Readiness | ✅ |