stage22: Abschluss & Production Readiness — E2E-Tests, CHANGELOG, Dokumentation
This commit is contained in:
NSCT Agent
155
README.md
155
README.md
@@ -1,6 +1,4 @@
|
||||
# ============================================================
|
||||
# NSCT — Neutral Search Crawler Tool
|
||||
# ============================================================
|
||||
|
||||
**NSCT** ist ein vollständig lokal betreibbares, containerisiertes Recherche- und
|
||||
Analyse-System. Es durchsucht Webquellen, extrahiert Inhalte, vergleicht
|
||||
@@ -9,30 +7,43 @@ quellengestützten Bericht.
|
||||
|
||||
**NSCT steht für:** Neutral Search Crawler Tool.
|
||||
|
||||
---
|
||||
|
||||
## Architektur-Übersicht
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌───────────────┐
|
||||
│ User / CLI │───▶│ NSCT API │───▶│ FastAPI / │
|
||||
│ │ │ (FastAPI) │ │ uvicorn │
|
||||
└─────────────┘ └──────────────┘ └───────────────┘
|
||||
│
|
||||
┌────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌───────────┐ ┌─────────┐ ┌──────────┐
|
||||
│ LLM │ │ Vision │ │ Audio │
|
||||
│ Provider │ │ Model │ │ Model │
|
||||
└───────────┘ └─────────┘ └──────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ PostgreSQL (evidence store) │
|
||||
└─────────────────────────────────────┘
|
||||
▲
|
||||
│
|
||||
┌──────────────┐
|
||||
│ SearXNG │ (optional search backend)
|
||||
└──────────────┘
|
||||
┌─────────────┐ ┌──────────────────────────┐ ┌───────────────┐
|
||||
│ User / CLI │───▶│ NSCT REST API (FastAPI) │───▶│ uvicorn │
|
||||
│ │ │ (async, background pip.)│ │ :8080 │
|
||||
└─────────────┘ └──────────┬───────────────┘ └───────────────┘
|
||||
│
|
||||
┌───────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Search │ │ Content │ │ Crawler │
|
||||
│ Providers │ │ Fetcher │ │ (SSRF-Prot) │
|
||||
│ (DDG, Multi) │ │ (HTTP/ │ │ (HTML,PDF, │
|
||||
└──────┬───────┘ │ Playwright)│ │ PDF, ...) │
|
||||
│ └──────┬─────┘ └──────────────┘
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Evidence Pipeline (Orchestrator) │
|
||||
│ Search → Fetch → Extract → Claim → │
|
||||
│ Compare → Score → Synthesis → Report │
|
||||
└──────────────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────┐
|
||||
│ PostgreSQL (asyncpg) │
|
||||
│ Evidence Store │
|
||||
└────────────────────────┘
|
||||
▲
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
│ LLM (Qwen3.6-35B) │
|
||||
│ Vision (Qwen2.5-VL-3B) │
|
||||
│ Audio (STT) │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
@@ -42,6 +53,7 @@ quellengestützten Bericht.
|
||||
```bash
|
||||
# 1. Kopiere die Beispiel-Env
|
||||
cp .env.example .env
|
||||
|
||||
# 2. Trage deine Endpunkte ein (LLM, Vision, Audio, PostgreSQL)
|
||||
|
||||
# 3. Starte alles
|
||||
@@ -59,40 +71,101 @@ Die API ist danach unter `http://localhost:8080` erreichbar.
|
||||
```
|
||||
nsct/
|
||||
├── src/nsct/
|
||||
│ ├── api/ # FastAPI-Routen
|
||||
│ ├── models/ # Pydantic-Schemata
|
||||
│ ├── providers/ # Abstrakte Provider-Interfaces
|
||||
│ ├── api/ # FastAPI-Routen (health, research, search, etc.)
|
||||
│ ├── agents/ # Research Planner & Validator
|
||||
│ ├── crawler/ # Content Fetcher, Extraction, Normalization
|
||||
│ ├── models/ # Pydantic v2 Schemas
|
||||
│ ├── orchestration/# State Machine, Budget, Gap Analysis
|
||||
│ ├── provenance.py # Vollständige Provenance Tracking
|
||||
│ ├── providers/ # LLM, Vision, Audio, Search, Multi, Priority
|
||||
│ ├── security/ # SSRF-Schutz, URL-Validierung
|
||||
│ └── storage/ # SQLAlchemy Models + Engine
|
||||
├── tests/ # pytest-Tests
|
||||
│ ├── stages/ # Pipeline-Stages (5-13: claim, evidence, synthesis)
|
||||
│ ├── storage/ # SQLAlchemy 2.0 Models + Engine
|
||||
│ ├── cli.py # Kommandozeileninterface
|
||||
│ ├── config.py # Pydantic BaseSettings Konfiguration
|
||||
│ ├── logging_config.py # Strukturiertes JSON-Logging
|
||||
│ ├── metrics.py # Prometheus Metriken
|
||||
│ └── main.py # FastAPI Application Factory
|
||||
├── tests/ # pytest-Tests (Unit + Integration)
|
||||
├── docker-compose.yml
|
||||
├── Dockerfile
|
||||
├── pyproject.toml
|
||||
├── .env.example
|
||||
├── README.md
|
||||
├── ARCHITECTURE.md
|
||||
├── SECURITY.md
|
||||
├── METHODOLOGY.md
|
||||
├── API.md
|
||||
├── DEPLOYMENT.md
|
||||
└── .env.example
|
||||
└── CHANGELOG.md
|
||||
```
|
||||
|
||||
## Schnellstart-Beispiel (CLI)
|
||||
|
||||
```bash
|
||||
nsct research "Welche wesentlichen Entwicklungen gab es im letzten Jahr bei Kernfusion?" \
|
||||
--depth normal \
|
||||
--language de
|
||||
```
|
||||
|
||||
## Schnellstart-Beispiel (API)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/research \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"query": "Welche Entwicklungen gab es bei Kernfusion?",
|
||||
"language": "de",
|
||||
"depth": "normal"
|
||||
}'
|
||||
```
|
||||
|
||||
Antwort:
|
||||
```json
|
||||
{
|
||||
"research_id": "uuid-...",
|
||||
"status": "pending",
|
||||
"query": "Welche Entwicklungen gab es bei Kernfusion?",
|
||||
"depth": "normal",
|
||||
"state": "created"
|
||||
}
|
||||
```
|
||||
|
||||
Den Status und das Ergebnis abrufen:
|
||||
```bash
|
||||
# Status
|
||||
curl http://localhost:8080/v1/research/<research_id>/status
|
||||
|
||||
# Bericht
|
||||
curl http://localhost:8080/v1/research/<research_id>/report
|
||||
|
||||
# Quellen
|
||||
curl http://localhost:8080/v1/research/<research_id>/sources
|
||||
|
||||
# Claims
|
||||
curl http://localhost:8080/v1/research/<research_id>/claims
|
||||
|
||||
# Evidence-Scores
|
||||
curl http://localhost:8080/v1/research/<research_id>/evidence
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
**Stage 0** — Repository und Architekturgrundlage.
|
||||
**Stage 22** — Abschluss & Production Readiness.
|
||||
|
||||
Stage 0 enthält:
|
||||
- Vollständige Projektstruktur mit allen Konfigurationsdateien
|
||||
- Pydantic v2 Datenmodelle
|
||||
- SQLAlchemy 2.0 Persistenzmodelle
|
||||
- Sicherheitshards (SSRF-Schutz, IP-Blocklist)
|
||||
- Provider-Interfaces (abstrakte Basisklassen)
|
||||
- Health-, Ready- und Provider-Endpunkte
|
||||
- Strukturiertes Logging
|
||||
- Docker-Konfiguration für PostgreSQL, SearXNG und NSCT
|
||||
Alle 22 Stages sind abgeschlossen. NSCT ist production-ready:
|
||||
|
||||
Die eigentliche Evidence-Pipeline (Search, Fetch, Extract, Claim, Compare, Report)
|
||||
wird in späteren Stages implementiert.
|
||||
- **Pipeline:** Search → Fetch → Extract → Claim → Compare → Score → Synthesize → Report
|
||||
- **State Machine:** 12 Zustände (CREATED → ... → COMPLETED/FAILED/CANCELLED)
|
||||
- **Budget Limits:** Hard limits für Queries, Sources, LLM-Requests, Duration
|
||||
- **Iterative Research:** Gap Analysis mit max 3 Runden
|
||||
- **REST API:** Vollständiger Research Lifecycle (POST/GET/DELETE)
|
||||
- **CLI:** Kommandozeileninterface mit depth, language, format Optionen
|
||||
- **Observability:** Strukturiertes Logging + Prometheus Metriken
|
||||
- **Docker Hardening:** Non-Root, Read-Only FS, Dropped Capabilities, Resource Limits
|
||||
- **Reproduzierbarkeit:** Vollständige Provenance + research_run_hash
|
||||
- **Neutralitäts-Tests:** Syndication, politische Aussagen, Dissens, Prompt Injection, fehlende Evidenz
|
||||
- **E2E-Tests:** 9 Integrationstests für den vollständigen Pipeline-Durchlauf
|
||||
|
||||
## Lizenz
|
||||
|
||||
|
||||
Reference in New Issue
Block a user