290 lines
9.6 KiB
Markdown
290 lines
9.6 KiB
Markdown
# Changelog — NSCT (Neutral Search Crawler Tool)
|
||
|
||
Alle signifikanten Änderungen an NSCT, organisiert nach Stage.
|
||
|
||
---
|
||
|
||
## [Unreleased]
|
||
|
||
### Stage 22 — Abschluss & Production Readiness
|
||
|
||
**Added**
|
||
- `tests/stages/test_e2e_integration.py` — 9 End-to-End-Integrationstests (POST → Pipeline → GET report)
|
||
- `CHANGELOG.md` — Vollständiges Versions-Changelog
|
||
|
||
**Documentation**
|
||
- `README.md` — Vollständig überarbeitet mit aktuellem Stand (Stage 0–22), Architektur-Diagramm, API-Referenz, Docker-Anleitung
|
||
- `ARCHITECTURE.md` — Aktualisierte Module-Beschreibungen mit allen Stages
|
||
- `API.md` — Vollständige Endpunkt-Dokumentation mit request/response Beispielen
|
||
- `DEPLOYMENT.md` — Deployment-Anleitung mit allen Konfigurationsoptionen
|
||
- `SECURITY.md` — Sicherheitsgrundsätze (Control/Evidence Plane, SSRF, Docker)
|
||
- `METHODOLOGY.md` — Methodik-Dokumentation: Neutralität, Quellenabhängigkeit, Claim-Vergleich
|
||
|
||
---
|
||
|
||
## Stage 21 — Reproduzierbarkeit
|
||
|
||
**Added**
|
||
- `src/nsct/provenance.py` — Vollständige Provenance aller Pipeline-Schritte
|
||
- `research_run_hash` im Report — deterministischer Hash aus Query, Sources, Claims
|
||
- Schema-Version Tracking für backward-compatible Reports
|
||
|
||
---
|
||
|
||
## Stage 20 — Context Budgeting
|
||
|
||
**Added**
|
||
- `src/nsct/orchestration/context_budget.py` — Pro-Stage Kontext-Limits
|
||
- Planner: 12k, Claim Extraction: 16k, Contradiction: 24k, Synthesis: 48k
|
||
- `_track_context_tokens()` im Orchestrator
|
||
- `tests/test_context_budget.py` — 18 Tests
|
||
- `tests/test_context_budget_integration.py` — 16 Integrationstests
|
||
- `src/nsct/providers/priority_queue.py` — async_fix für Python 3.14+
|
||
|
||
---
|
||
|
||
## Stage 19 — Performanceoptimierung
|
||
|
||
**Added**
|
||
- `src/nsct/providers/semaphore.py` — LLM Concurrency Semaphore (configurable, default: 3)
|
||
- `src/nsct/providers/priority_queue.py` — Priority queue mit HIGH/NORMAL/LOW Prioritäten
|
||
- `tests/test_priority_limiter.py` — 14 Tests für Priority Limiter
|
||
|
||
---
|
||
|
||
## Stage 18 — Docker Hardening
|
||
|
||
**Added**
|
||
- Non-Root-User (`nsct`, UID 1000) im Dockerfile
|
||
- Read-Only-Filesystem mit tmpfs
|
||
- Dropped Capabilities (`cap_drop: [ALL]`)
|
||
- Resource Limits (Memory, CPU) per Container
|
||
- Healthcheck für Docker Container
|
||
|
||
---
|
||
|
||
## Stage 17 — Neutralitäts-Tests
|
||
|
||
**Added**
|
||
- `tests/stages/test_neutrality_a_syndication.py` — Syndication-Erkennung
|
||
- `tests/stages/test_neutrality_b_political_statements.py` — Politische Aussagen
|
||
- `tests/stages/test_neutrality_c_scientific_disagreement.py` — Wissenschaftlicher Dissens
|
||
- `tests/stages/test_neutrality_d_prompt_injection.py` — Prompt Injection
|
||
- `tests/stages/test_neutrality_e_missing_evidence.py` — Fehlende Evidenz
|
||
|
||
---
|
||
|
||
## Stage 16 — Observability
|
||
|
||
**Added**
|
||
- `src/nsct/logging_config.py` — Strukturiertes JSON-Logging mit request context
|
||
- `src/nsct/metrics.py` — Prometheus Metriken (search, sources, claims, contradictions, LLM)
|
||
- `tests/test_logging.py` — Logging-Tests
|
||
- `tests/test_metrics.py` — Metriken-Tests
|
||
|
||
---
|
||
|
||
## Stage 15 — CLI
|
||
|
||
**Added**
|
||
- `src/nsct/cli.py` — Kommandozeileninterface
|
||
- `tests/test_cli.py` — CLI-Tests
|
||
|
||
---
|
||
|
||
## Stage 14 — REST API
|
||
|
||
**Added**
|
||
- `src/nsct/api/rest_research.py` — Vollständige Research REST API
|
||
- `POST /v1/research` — Start research (async, background pipeline)
|
||
- `GET /v1/research` — Liste aller Recherchen (paginiert)
|
||
- `GET /v1/research/{id}` — Metadaten
|
||
- `GET /v1/research/{id}/status` — State-Machine Status
|
||
- `GET /v1/research/{id}/sources` — Quellen
|
||
- `GET /v1/research/{id}/claims` — Claims
|
||
- `GET /v1/research/{id}/evidence` — Evidence-Scores
|
||
- `GET /v1/research/{id}/report` — Synthese-Bericht
|
||
- `DELETE /v1/research/{id}` — Löschen (nicht-komplettiert)
|
||
- `tests/test_rest_research.py` — Komplette API-Tests
|
||
- Depth-Configs: `quick`, `normal`, `deep` mit escalating Budgets
|
||
|
||
---
|
||
|
||
## Stage 13 — Iterative Research / Gap Analysis
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage13_gap_analysis.py` — Lückenerkennung
|
||
- `src/nsct/orchestration/gap_analysis.py` — GapSearchQuery
|
||
- Max 3 iterative Runden (konfigurierbar über `NSCT_MAX_RESEARCH_ROUNDS`)
|
||
- `tests/test_stage13_gap_analysis.py` — 17 Tests
|
||
|
||
---
|
||
|
||
## Stage 12 — Research Orchestrator
|
||
|
||
**Added**
|
||
- `src/nsct/orchestration/state.py` — State Machine (12 Zustände)
|
||
- `src/nsct/orchestration/budget.py` — Hard Budget Limits (Pydantic, frozen)
|
||
- `src/nsct/orchestration/models.py` — ResearchRun Pydantic Model (frozen)
|
||
- `src/nsct/orchestration/orchestrator.py` — Vollständige Pipeline-Steuerung
|
||
|
||
---
|
||
|
||
## Stage 11 — Audio Integration
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage11_audio.py` — STT-Transkription mit timestamped Claims
|
||
- `src/nsct/models/audio.py` — Audio Pydantic Schema
|
||
- `src/nsct/providers/audio.py` — AudioProvider Implementation
|
||
- `src/nsct/api/audio.py` — Audio API Endpoint
|
||
- `tests/stages/test_stage11_audio.py` — ~20 Tests
|
||
|
||
---
|
||
|
||
## Stage 10 — Vision Integration
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage10_vision.py` — Qwen2.5-VL-3B Integration
|
||
- `src/nsct/models/vision.py` — Vision Pydantic Schema
|
||
- `src/nsct/providers/vision.py` — VisionProvider Implementation
|
||
- `src/nsct/api/vision.py` — Vision API Endpoint
|
||
- `tests/stages/test_stage10_vision.py` — ~25 Tests
|
||
|
||
---
|
||
|
||
## Stage 9 — Neutral Synthesis Engine
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage9_synthesis.py` — LLM-basierte neutrale Berichts-Synthese
|
||
- `src/nsct/models/synthesis.py` — Synthese Pydantic Schema
|
||
- `src/nsct/api/synthesis.py` — Synthese API Endpoint
|
||
- `tests/stages/test_stage9_synthesis.py` — 25 Tests
|
||
|
||
---
|
||
|
||
## Stage 8 — Evidence Scoring
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage8_evidence_scoring.py` — 6-dimensionale Evidence-Scores
|
||
- `evidence_type` und `raw_scores_json` Felder
|
||
- `src/nsct/api/stage8.py` — Evidence Scoring API
|
||
- `tests/stages/test_stage8_evidence_scoring.py` — ~60 Tests
|
||
|
||
---
|
||
|
||
## Stage 7 — Claim Clustering & Contradiction
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage7_clustering.py` — LLM-Clustering
|
||
- `src/nsct/stages/stage7_normalize_numerics.py` — Numerische Normalisierung
|
||
- `tests/stages/test_stage7_clustering.py` — 79 Tests
|
||
|
||
---
|
||
|
||
## Stage 6 — Source Independence & Citation Graph
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage6_source_independence.py` — Syndication-Erkennung
|
||
- `src/nsct/models/source_independence.py` — Independence Pydantic Schema
|
||
- `tests/stages/test_stage6_source_independence.py` — 43 Tests
|
||
|
||
---
|
||
|
||
## Stage 5 — Claim Extraction
|
||
|
||
**Added**
|
||
- `src/nsct/stages/stage5_extract_claims.py` — Atomare, überprüfbare Claims
|
||
- `src/nsct/models/claim.py` — Claim Pydantic Schema (mit provenance)
|
||
- `tests/stages/test_stage5_extract_claims.py` — 36 Tests
|
||
|
||
---
|
||
|
||
## Stage 4 — Research Planner
|
||
|
||
**Added**
|
||
- `src/nsct/agents/planner.py` — LLM-basierte Recherchestrategie
|
||
- `src/nsct/models/plan.py` — Plan Pydantic Schema
|
||
- `src/nsct/api/planner.py` — Planner API Endpoint
|
||
- `tests/test_planner.py` — 25 Tests
|
||
|
||
---
|
||
|
||
## Stage 3 — Crawler & Content Extraction
|
||
|
||
**Added**
|
||
- `src/nsct/crawler/fetcher.py` — Asynchroner HTTP-Fetcher mit SSRF-Schutz
|
||
- `src/nsct/crawler/extraction.py` — Content-Extraktion (HTML, Text, PDF)
|
||
- `src/nsct/crawler/manager.py` — Batch-Verwaltung
|
||
- `src/nsct/crawler/normalize.py` — Dokumenten-Normalisierung
|
||
- `src/nsct/crawler/pdf.py` — PDF-Extraktion
|
||
- `src/nsct/crawler/policy.py` — SSRF-Schutz Policy
|
||
- `tests/test_crawler.py` — ~15 Tests
|
||
|
||
---
|
||
|
||
## Stage 2 — Search Provider Abstraction
|
||
|
||
**Added**
|
||
- `src/nsct/providers/duckduckgo.py` — DuckDuckGo Search Provider
|
||
- `src/nsct/providers/multi.py` — MultiProviderSearch (parallele Abfrage)
|
||
- `src/nsct/api/search.py` — Search API Endpoint
|
||
- `tests/test_search.py` — 25 Tests
|
||
|
||
---
|
||
|
||
## Stage 1 — Model Provider Layer
|
||
|
||
**Added**
|
||
- `src/nsct/providers/llm.py` — OpenAI-kompatibler LLM-Provider
|
||
- `src/nsct/providers/abstract.py` — Abstrakte Provider-Basisklassen
|
||
- `src/nsct/api/debug.py` — Debug Endpoints (NSCT_DEBUG only)
|
||
- `tests/test_crawler.py`, `tests/test_health.py`, `tests/test_planner.py` — Tests für Stage 1
|
||
|
||
---
|
||
|
||
## Stage 0 — Repository & Architekturgrundlage
|
||
|
||
**Added**
|
||
- Vollständige Projektstruktur (`src/nsct/` mit allen Unterverzeichnissen)
|
||
- `pyproject.toml` — Python Project Configuration
|
||
- `Dockerfile` — Container Image
|
||
- `docker-compose.yml` — PostgreSQL, SearXNG, NSCT
|
||
- `.env.example` — Konfigurations-Vorlage
|
||
- `src/nsct/config.py` — Pydantic BaseSettings Konfiguration
|
||
- `src/nsct/storage/` — SQLAlchemy 2.0 Models + Engine (asyncpg)
|
||
- `src/nsct/models/schemas.py` — Pydantic v2 Schemas
|
||
- `src/nsct/security/policy.py` — SSRF-Schutz, URL-Validierung, IP-Blocklist
|
||
- `src/nsct/api/health.py` — `/health`, `/ready`, `/providers` Endpunkte
|
||
- `tests/test_health.py` — Health Check Tests
|
||
- `README.md`, `ARCHITECTURE.md`, `SECURITY.md`, `METHODOLOGY.md` — Grundlegende Dokumentation
|
||
- 6 Basis-Tests
|
||
|
||
---
|
||
|
||
**Zusammenfassung:**
|
||
|
||
| Stage | Description | Status |
|
||
|-------|-------------|--------|
|
||
| 0 | Repository & Architekturgrundlage | ✅ |
|
||
| 1 | Model Provider Layer | ✅ |
|
||
| 2 | Search Provider Abstraction | ✅ |
|
||
| 3 | Crawler & Content Extraction | ✅ |
|
||
| 4 | Research Planner | ✅ |
|
||
| 5 | Claim Extraction | ✅ |
|
||
| 6 | Source Independence & Citation Graph | ✅ |
|
||
| 7 | Claim Clustering & Contradiction | ✅ |
|
||
| 8 | Evidence Scoring | ✅ |
|
||
| 9 | Neutral Synthesis Engine | ✅ |
|
||
| 10 | Vision Integration | ✅ |
|
||
| 11 | Audio Integration | ✅ |
|
||
| 12 | Research Orchestrator | ✅ |
|
||
| 13 | Iterative Research / Gap Analysis | ✅ |
|
||
| 14 | REST API | ✅ |
|
||
| 15 | CLI | ✅ |
|
||
| 16 | Observability | ✅ |
|
||
| 17 | Neutralitäts-Tests | ✅ |
|
||
| 18 | Docker Hardening | ✅ |
|
||
| 19 | Performanceoptimierung | ✅ |
|
||
| 20 | Context Budgeting | ✅ |
|
||
| 21 | Reproduzierbarkeit | ✅ |
|
||
| 22 | Abschluss & Production Readiness | ✅ | |