From 8276975e93f9fe4eb1012a4ae64efa6935e8548c Mon Sep 17 00:00:00 2001 From: NSCT Agent Date: Sun, 6 Sep 2026 08:37:06 +0000 Subject: [PATCH] Admin Handbook: Backend (NSCT API) --- ADMIN-Backend.md | 1214 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1214 insertions(+) create mode 100644 ADMIN-Backend.md diff --git a/ADMIN-Backend.md b/ADMIN-Backend.md new file mode 100644 index 0000000..574c409 --- /dev/null +++ b/ADMIN-Backend.md @@ -0,0 +1,1214 @@ +# ADMIN-Handbook — NSCT Backend (API) + +Produziert vom NSCT Development Team — Stand: 2025-09-06 + +--- + +## 1. Übersicht + +**NSCT** (Neutral Search Crawler Tool) ist ein lokal betreibbares, containerisiertes Recherche- und Analyse-System. Es durchsucht automatisch Webquellen, extrahiert Behauptungen (Claims), bewertet Evidenz multi-dimensional und generiert neutrale, quellengestützte Berichte. + +### Architektur-Überblick + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Docker Compose Stack │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ nsct-api │────▶│ postgres │────▶│ searxng │ │ +│ │ :8080 │ │ :5432 │ │ :8888 │ │ +│ │ FastAPI │ │ PG 16 │ │ SearXNG │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ User ───▶ LLM-Provider (:8030) (extern, z.B. Ollama/VLLM) │ +│ User ───▶ Audio-Provider (:8030) (extern, STT/TTS) │ +│ User ───▶ Vision-Provider (:8030) (extern, Bildanalyse) │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +**Kernprinzipien:** +- Container-first — alles läuft in Docker Compose +- Environment-only Configuration — keine Config-Dateien, alles über `.env` +- Security-first — SSRF-Schutz, non-root, read-only FS, dropped capabilities +- Provenance-Pflicht — jeder Schritt ist nachverfolgbar und reproduzierbar + +**Module:** + +| Modul | Path | Beschreibung | +|-------|------|-------------| +| API Layer | `src/nsct/api/` | REST-Endpunkte (FastAPI) | +| Models | `src/nsct/models/` | Pydantic v2 Schemas (DTOs) | +| Storage | `src/nsct/storage/` | SQLAlchemy 2.0 + asyncpg | +| Providers | `src/nsct/providers/` | LLM/Vision/Audio/Search Abstraktionen | +| Security | `src/nsct/security/` | SSRF-Schutz, URL-Validierung | +| Config | `src/nsct/config.py` | Environment-only, Pydantic BaseSettings | +| Logging | `src/nsct/logging_config.py` | JSON-Formatter, per-request context | +| Metrics | `src/nsct/metrics.py` | Prometheus-kompatible Metriken | +| Orchestration | `src/nsct/orchestration/` | State-Machine, Budget-Limits | +| Stages | `src/nsct/stages/` | Pipeline-Stages 5–13 | +| Provenance | `src/nsct/provenance.py` | Deterministische Hashes, Audit-Trail | + +--- + +## 2. Schnellstart + +### 2.1 Voraussetzungen + +- Docker ≥ 24.0 +- Docker Compose ≥ 2.23 +- Mindestens 2 GB freier RAM +- Zugang zu einem OpenAI-kompatiblen LLM-Endpunkt (Port 8030) +- Git installiert + +### 2.2 Repository klonen + +```bash +git clone https://github.com/faligam/nsct.git +cd nsct +``` + +### 2.3 Environment-Datei erstellen und konfigurieren + +```bash +cp .env.example .env +``` + +Trage die folgenden Werte in `.env` ein: + +```env +# ---- LLM (Hauptmodell, verpflichtend) ---- +NSCT_LLM_BASE_URL=http://:8030/openai/v1 +NSCT_LLM_MODEL=Qwen3.6-35B +NSCT_LLM_MAX_CONCURRENCY=3 +NSCT_LLM_API_KEY= + +# ---- Vision (Bildanalyse, optional) ---- +NSCT_VISION_BASE_URL=http://:8030/openai/visual/v1 +NSCT_VISION_MODEL=Qwen2.5-VL-3B +NSCT_VISION_API_KEY= + +# ---- Audio (STT/TTS, optional) ---- +NSCT_AUDIO_BASE_URL=http://:8030/hermes-audio +NSCT_AUDIO_MODEL=default +NSCT_AUDIO_API_KEY= + +# ---- PostgreSQL ---- +POSTGRES_USER=nsct +POSTGRES_PASSWORD= +POSTGRES_DB=nsct +NSCT_DB_URL=postgresql+asyncpg://nsct:@postgres:5432/nsct + +# ---- SearXNG (optional, interne Suche) ---- +NSCT_SEARXNG_BASE_URL=http://searxng:8080 + +# ---- General ---- +NSCT_DEBUG=false +``` + +**Umgebungsvariablen-Referenz:** + +| Variable | Typ | Default | Beschreibung | +|----------|-----|---------|-------------| +| `NSCT_LLM_BASE_URL` | string | `""` | OpenAI-kompatibler LLM-Endpunkt | +| `NSCT_LLM_MODEL` | string | `""` | Modell-Name | +| `NSCT_LLM_MAX_CONCURRENCY` | int | `3` | Max parallele LLM-Requests | +| `NSCT_LLM_API_KEY` | string | `""` | API-Key für LLM-Provider | +| `NSCT_VISION_BASE_URL` | string | `""` | Vision-Endpunkt | +| `NSCT_VISION_MODEL` | string | `""` | Vision-Modell-Name | +| `NSCT_VISION_API_KEY` | string | `""` | API-Key für Vision | +| `NSCT_AUDIO_BASE_URL` | string | `""` | Audio-Endpunkt (STT/TTS) | +| `NSCT_AUDIO_MODEL` | string | `default` | Audio-Modell | +| `NSCT_AUDIO_API_KEY` | string | `""` | API-Key für Audio | +| `POSTGRES_USER` | string | `nsct` | PostgreSQL-User | +| `POSTGRES_PASSWORD` | string | `nsct_secret` | PostgreSQL-Passwort | +| `POSTGRES_DB` | string | `nsct` | PostgreSQL-Name | +| `NSCT_DB_URL` | string | `""` | DB-Connection-String | +| `NSCT_SEARXNG_BASE_URL` | string | `""` | SearXNG-Instance-URL | +| `NSCT_DEBUG` | bool | `false` | Debug-Modus (Swagger aktiv) | +| `NSCT_LOG_LEVEL` | string | `INFO` | Log-Level (DEBUG/INFO/WARNING/ERROR/CRITICAL) | +| `NSCT_METRICS_ENABLED` | bool | `true` | `/metrics` Endpoint aktivieren | + +### 2.4 Dienste starten + +```bash +# Vollständiger Stack mit Build +docker compose up --build -d + +# Oder nur API + PostgreSQL (ohne SearXNG) +docker compose up --build -d nsct-api postgres +``` + +### 2.5 Health-Check prüfen + +```bash +# Grundlegender Liveness-Check (immer ok, wenn Prozess lebt) +curl -s http://localhost:8080/health | jq + +# Readiness-Check (prüft LLM-Erreichbarkeit) +curl -s http://localhost:8080/ready | jq + +# Provider-Status (zeigt konfigurierte Models) +curl -s http://localhost:8080/providers | jq +``` + +**Erwartete Antwort `/health`:** +```json +{ + "status": "ok", + "version": "0.1.0" +} +``` + +**Erwartete Antwort `/ready` (bei voller Konfiguration):** +```json +{ + "status": "ready", + "llm": "ok", + "database": "unknown", + "llm_model": "Qwen3.6-35B" +} +``` + +### 2.6 curl-Tests + +```bash +# Provider-Status anzeigen +curl -s http://localhost:8080/providers | jq + +# Prometheus-Metriken (falls aktiv) +curl -s http://localhost:8080/metrics + +# Swagger UI (nur bei NSCT_DEBUG=true) +# http://localhost:8080/docs +# ReDoc: http://localhost:8080/redoc +``` + +--- + +## 3. PostgreSQL Setup + +### 3.1 Verbindung einrichten + +Die Standard-Verbindungsdaten aus `.env`: + +``` +Host: postgres (intern im Docker-Netzwerk) +Port: 5432 +Database: nsct +User: nsct +Password: +``` + +Shell-Zugriff auf die DB: + +```bash +# In den Container verbinden +docker compose exec postgres psql -U nsct -d nsct + +# Oder von der Host-Shell (wenn Port 5432 gemappt ist) +PGPASSWORD= psql -h localhost -U nsct -d nsct -p 5432 +``` + +### 3.2 Backup (pg_dump) + +```bash +# Vollständiges Dump (SQL-Format) +docker compose exec postgres pg_dump -U nsct -d nsct > nsct_backup_$(date +%Y%m%d).sql + +# Komprimiertes Dump +docker compose exec postgres pg_dump -U nsct -d nsct | gzip > nsct_backup_$(date +%Y%m%d).sql.gz + +# Nur Struktur (ohne Daten) +docker compose exec postgres pg_dump -U nsct -d nsct --schema-only > nsct_schema.sql + +# Nur Daten (ohne Struktur) +docker compose exec postgres pg_dump -U nsct -d nsct --data-only > nsct_data.sql + +# Backup in ein eigenes Volume (für Offline-Storage) +docker compose exec postgres pg_dump -U nsct -d nsct > /tmp/nsct_backup.sql +docker cp nsct-postgres:/tmp/nsct_backup.sql ./nsct_backup.sql +``` + +### 3.3 Restore (pg_restore) + +```bash +# Restore aus SQL-Datei (psql-Format) +docker compose exec -T postgres psql -U nsct -d nsct < nsct_backup.sql + +# Restore aus komprimierter Datei +gunzip -c nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct + +# Restore aus pg_dumpall (wenn mit pg_dumpall gesichert) +docker compose exec -T postgres psql -U nsct -d nsct < nsct_full_backup.sql +``` + +### 3.4 Migrationen + +Die Datenbank wird aktuell **manuell** über SQLAlchemy-Alchemy-Reflection initialisiert. Migrationen werden über Alembic in zukünftigen Stages implementiert. + +```bash +# Aktuelle Tabellenliste +docker compose exec postgres psql -U nsct -d nsct -c "\dt" + +# Tabellenschema anzeigen +docker compose exec postgres psql -U nsct -d nsct -c "\d+ sources" + +# Datenbankgröße +docker compose exec postgres psql -U nsct -d nsct -c "SELECT pg_database_size('nsct');" + +# Tabellen-Größen (Byte) +docker compose exec postgres psql -U nsct -d nsct -c "\l+" +docker compose exec postgres psql -U nsct -d nsct -c "SELECT relname, pg_size_pretty(pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename))) FROM pg_tables WHERE schemaname = 'public';" +``` + +### 3.5 Troubleshooting + +```bash +# PostgreSQL ist nicht bereit — Connection refused +# Lösung: Warte bis health check bestanden ist +docker compose ps postgres + +# Prüfe ob Container läuft +docker compose logs postgres + +# Manueller health-check +docker compose exec postgres pg_isready -U nsct -d nsct + +# Wenn Password-Auth fehlschlägt: +# .env prüfen: POSTGRES_PASSWORD muss mit dem Wert übereinstimmen, +# den du in NSCT_DB_URL verwendest. + +# Connection-String validieren: +echo "$NSCT_DB_URL" | python3 -c "import sys; print(sys.stdin.read())" + +# PostgreSQL-Fehler in Container-Logs +docker compose logs --tail=50 postgres +``` + +**Häufige Fehler:** + +| Fehler | Ursache | Lösung | +|--------|---------|--------| +| `FATAL: database "nsct" does not exist` | `POSTGRES_DB` in `.env` falsch gesetzt | Prüfe `POSTGRES_DB` und `NSCT_DB_URL` | +| `FATAL: password authentication failed` | Password mismatch | `.env` korrigieren, `docker compose down -v && up` | +| `could not connect to server: Connection refused` | PostgreSQL noch nicht ready | Warte, prüfe `docker compose ps` | +| `FATAL: too many connections` | Pool exhausted (default: 100) | `max_connections` erhöhen oder Poolgröße senken | +| `relation "xxx" does not exist` | Tabellen nicht initialisiert | Prüfe, ob API korrekt gestartet ist | + +--- + +## 4. LLM-Endpunkte konfigurieren + +Das NSCT-Backend erwartet OpenAI-kompatible Endpunkte (Standard: Ollama, vLLM, LMStudio). + +### 4.1 Qwen3.6-35B (Hauptmodell) + +Das primäre LLM für Claim-Extraktion, Synthese, Evidence-Scoring und alle LLM-gesteuerten Stages. + +```env +NSCT_LLM_BASE_URL=http://:8030/openai/v1 +NSCT_LLM_MODEL=Qwen3.6-35B +NSCT_LLM_MAX_CONCURRENCY=3 +NSCT_LLM_API_KEY= +``` + +**Endpoint-Validierung:** + +```bash +# Modellliste abfragen +curl -s http://:8030/openai/v1/models | jq + +# Erwartete Antwort: +# { +# "data": [ +# { "id": "Qwen3.6-35B", "object": "model", "owned_by": "qwen" } +# ] +# } + +# Chat-Completion testen +curl -s http://:8030/openai/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "model": "Qwen3.6-35B", + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 10 + }' | jq +``` + +### 4.2 Qwen2.5-VL-3B (Vision) + +Für Bild- und Dokumentenanalyse (Screenshots, PDFs, Charts). + +```env +NSCT_VISION_BASE_URL=http://:8030/openai/visual/v1 +NSCT_VISION_MODEL=Qwen2.5-VL-3B +NSCT_VISION_API_KEY= +``` + +**Endpoint-Validierung:** + +```bash +# Models-Endpoint +curl -s http://:8030/openai/visual/v1/models | jq + +# Chat mit Bild (Base64) testen +curl -s http://:8030/openai/visual/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "model": "Qwen2.5-VL-3B", + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, + {"type": "text", "text": "Was siehst du?"} + ] + } + ] + }' | jq +``` + +### 4.3 Audio-Modell (STT/TTS) + +Für Audio-Transkription (Stage 11) und Text-to-Speech. + +```env +NSCT_AUDIO_BASE_URL=http://:8030/hermes-audio +NSCT_AUDIO_MODEL=default +NSCT_AUDIO_API_KEY= +``` + +**Endpoint-Validierung:** + +```bash +# Test-Request (STT) +curl -s http://:8030/hermes-audio/transcribe \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "model": "default", + "language": "de" + }' | jq +``` + +### 4.4 Endpoint-Validierung aus der API + +```bash +# Alle Provider auf einmal prüfen +curl -s http://localhost:8080/providers | jq + +# Readiness-Check (LLM-Erreichbarkeit) +curl -s http://localhost:8080/ready | jq +``` + +### 4.5 Troubleshooting + +| Fehler | Ursache | Lösung | +|--------|---------|--------| +| `LLM connectivity check failed` | Endpunkt nicht erreichbar | Prüfe Netzwerk, Firewall, `.env` | +| `error:ConnectionRefused` | LLM-Server down | Starte LLM-Server neu | +| `error:401` | API-Key falsch | `.env` prüfen, Key regenerieren | +| `error:404` | Falscher Endpoint-Path | Prüfe Base-URL (z.B. `/openai/v1` vs `/v1`) | +| `timeout` | LLM zu langsam | `NSCT_LLM_MAX_CONCURRENCY` senken, Timeout erhöhen | + +**LLM-Server-Status von innen prüfen:** + +```bash +# Vom nsct-api Container aus zum LLM-Host ping'en +docker compose run --rm nsct-api curl -s http://:8030/openai/v1/models | jq +``` + +--- + +## 5. API-Key-Verwaltung + +> **Hinweis:** In der aktuellen Version (Stage 22+) ist die Authentifizierung noch nicht implementiert. Die Struktur ist vorbereitet für zukünftige API-Key-Authentifizierung. + +### 5.1 User anlegen (vorbereitete SQL-Struktur) + +Die Datenbank-Struktur für User/API-Key-Management: + +```sql +-- User-Tabelle (vorbereitet für Stage 22+) +CREATE TABLE IF NOT EXISTS users ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + username VARCHAR(255) UNIQUE NOT NULL, + email VARCHAR(255) UNIQUE, + is_active BOOLEAN DEFAULT TRUE, + created_at TIMESTAMP DEFAULT NOW(), + updated_at TIMESTAMP DEFAULT NOW() +); + +-- API-Key-Tabelle +CREATE TABLE IF NOT EXISTS api_keys ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID REFERENCES users(id) ON DELETE CASCADE, + key_hash VARCHAR(255) NOT NULL, + name VARCHAR(255), + is_active BOOLEAN DEFAULT TRUE, + created_at TIMESTAMP DEFAULT NOW(), + expires_at TIMESTAMP, + last_used_at TIMESTAMP +); + +-- Indexes +CREATE INDEX idx_api_keys_key_hash ON api_keys(key_hash); +CREATE INDEX idx_api_keys_user_id ON api_keys(user_id); +CREATE INDEX idx_api_keys_is_active ON api_keys(is_active); +``` + +### 5.2 User und API-Key anlegen + +```sql +-- User erstellen +INSERT INTO users (username, email, is_active) +VALUES ('admin', 'admin@example.com', TRUE) +ON CONFLICT (username) DO NOTHING; + +-- API-Key generieren (Hash speichern) +-- 1. Generiere einen sicheren Key (von der Admin-Shell) +python3 -c "import secrets, hashlib; key = secrets.token_hex(32); print('KEY:', key); print('HASH:', hashlib.sha256(key.encode()).hexdigest())" + +-- 2. Hash in der DB speichern (behalte den Original-Key sicher!) +INSERT INTO api_keys (user_id, key_hash, name, is_active) +SELECT u.id, '', 'admin-key', TRUE +FROM users u WHERE u.username = 'admin'; +``` + +### 5.3 API-Key-Hashes verwalten + +```sql +-- Alle aktiven Keys mit User +SELECT ak.name, ak.is_active, ak.created_at, ak.expires_at, u.username, u.email +FROM api_keys ak +JOIN users u ON ak.user_id = u.id +WHERE ak.is_active = TRUE +ORDER BY ak.created_at DESC; + +-- Key-Verwendung (last_used_at) +UPDATE api_keys SET last_used_at = NOW() WHERE key_hash = ''; + +-- Ablaufende Keys prüfen +SELECT ak.name, ak.expires_at, u.username +FROM api_keys ak +JOIN users u ON ak.user_id = u.id +WHERE ak.expires_at < NOW() AND ak.is_active = TRUE; + +-- Inaktive Keys deaktivieren +UPDATE api_keys SET is_active = FALSE WHERE is_active = FALSE; +``` + +### 5.4 Debugging von Auth-Problemen + +```sql +-- API-Key existiert und ist aktiv? +SELECT * FROM api_keys WHERE key_hash = '' AND is_active = TRUE; + +-- User existiert? +SELECT * FROM users WHERE username = ''; + +-- Hash-Korrelation prüfen +SELECT ak.key_hash, u.username, ak.is_active +FROM api_keys ak +JOIN users u ON ak.user_id = u.id +WHERE ak.key_hash = ''; +``` + +--- + +## 6. Docker Compose Konfiguration + +### 6.1 docker-compose.yml erklärt + +```yaml +# NSCT — docker-compose.yml +# Services: nsct-api, postgres, optional searxng +``` + +### 6.2 Services + +| Service | Image | Port | Beschreibung | +|---------|-------|------|-------------| +| `nsct-api` | `build: .` | `8080:8080` | FastAPI-Server | +| `postgres` | `postgres:16-alpine` | `5432:5432` | PostgreSQL 16 | +| `searxng` | `searxng/searxng:latest` | `8888:8080` | Interne Suche (optional) | + +### 6.3 Service-Details + +#### nsct-api + +```yaml +nsct-api: + build: + context: . + dockerfile: Dockerfile + container_name: nsct-api + ports: + - "8080:8080" + environment: + NSCT_DB_URL: postgresql+asyncpg://...@postgres:5432/nsct + NSCT_LLM_MAX_CONCURRENCY: "3" + depends_on: + postgres: + condition: service_healthy + searxng: + condition: service_started + volumes: + - nsct_data:/app/data + healthcheck: + test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8080/health')\" || exit 1"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 15s + read_only: true + tmpfs: + - /tmp + cap_drop: + - ALL +``` + +**Umgebungsvariablen im Container:** +- `NSCT_DB_URL` — DB-URL (Host: `postgres`) +- `NSCT_LLM_MAX_CONCURRENCY` — LLM-Concurrency (erbt aus `.env`) +- Alle anderen vars aus `.env` (`env_file: .env`) + +#### postgres + +```yaml +postgres: + image: postgres:16-alpine + container_name: nsct-postgres + environment: + POSTGRES_USER: ${POSTGRES_USER:-nsct} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-nsct_secret} + POSTGRES_DB: ${POSTGRES_DB:-nsct} + ports: + - "5432:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-nsct} -d ${POSTGRES_DB:-nsct}"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 10s + deploy: + resources: + limits: + memory: 512M + cpus: "0.5" +``` + +#### searxng (optional) + +```yaml +searxng: + image: searxng/searxng:latest + container_name: nsct-searxng + ports: + - "8888:8080" + volumes: + - searxng_settings:/etc/searxng + environment: + - SEARXNG_BASE_URL=http://localhost:8888/ + - SEARXNG_SECRET_KEY=nsct_searxng_secret_key_change_me + read_only: true + tmpfs: + - /tmp + cap_drop: + - ALL +``` + +### 6.4 Volumes + +| Volume | Pfad | Beschreibung | +|--------|------|-------------| +| `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL-Datenbankdateien | +| `nsct_data` | `/app/data` | NSCT-Runtime-Daten (Cache, etc.) | +| `searxng_settings` | `/etc/searxng` | SearXNG-Konfiguration | + +### 6.5 Networks + +Standard `docker compose` Netzwerk (bridge). Alle Services sind über Service-Namen erreichbar: + +``` +postgres → postgres:5432 +searxng → searxng:8080 +nsct-api → nsct-api:8080 +``` + +Für isolierte Netzwerke: + +```yaml +networks: + nsct-internal: + driver: bridge +``` + +### 6.6 Resource Limits + +| Service | Memory Limit | CPU Limit | Memory Reservation | +|---------|-------------|-----------|-------------------| +| nsct-api | 1G | 1.0 | 256M | +| postgres | 512M | 0.5 | — | +| searxng | 512M | 0.5 | — | + +**Anpassen:** In `docker-compose.yml` unter `deploy.resources.limits` editieren. + +### 6.7 Health Checks + +| Service | Endpoint | Interval | Timeout | Retries | +|---------|----------|----------|---------|---------| +| nsct-api | `GET /health` | 30s | 5s | 3 | +| postgres | `pg_isready` | 10s | 5s | 5 | + +--- + +## 7. Health-Check, Logs, Troubleshooting + +### 7.1 curl-Befehle + +```bash +# Basic health — Liveness +curl -s http://localhost:8080/health | jq + +# Readiness — prüft LLM-Connectivity +curl -s http://localhost:8080/ready | jq + +# Provider-Info +curl -s http://localhost:8080/providers | jq + +# Prometheus-Metriken +curl -s http://localhost:8080/metrics + +# Swagger Docs (nur bei NSCT_DEBUG=true) +curl -s -I http://localhost:8080/docs +``` + +### 7.2 Docker Logs + +```bash +# Alle Logs (live) +docker compose logs -f + +# Nur API-Container +docker compose logs -f nsct-api + +# Letzte 100 Zeilen +docker compose logs --tail=100 nsct-api + +# Errors nur +docker compose logs -f nsct-api | grep ERROR + +# PostgreSQL Logs +docker compose logs -f postgres + +# Logs mit Zeitstempeln +docker compose logs -f --timestamp nsct-api +``` + +### 7.3 Container-Status prüfen + +```bash +# Container-Status +docker compose ps + +# Container-Details (JSON) +docker inspect nsct-api | jq '.[0].State' + +# Container-Logs (letzte 50 Zeilen, nicht-live) +docker compose logs --tail=50 nsct-api + +# Resource-Verbrauch +docker stats nsct-api nsct-postgres nsct-searxng +``` + +### 7.4 Restart-Prozedur + +```bash +# Soft restart (alle Services) +docker compose restart + +# Nur API neu starten +docker compose restart nsct-api + +# Vollständiger Reload (neuer Build) +docker compose down +docker compose up --build -d + +# Container komplett entfernen und neu erstellen +docker compose down -v +docker compose up --build -d +``` + +### 7.5 Common Errors und Lösungen + +| Error | Lösung | +|-------|--------| +| `Connection refused` auf `/health` | Warte 15s (start_period), prüfe `docker compose ps` | +| `LLM: error:Connection refused` | LLM-Server prüfen, `.env` Base-URL validieren | +| `LLM: error:ConnectionRefused` | Netzwerk-Verbindung prüfen: `docker compose run --rm nsct-api curl -v ` | +| `HTTP 503` bei Research-Requests | Readiness prüfen: `curl localhost:8080/ready` | +| `no space left on device` | `docker system prune -f` ausführen | +| Container OOMKilled | Memory-Limit in `docker-compose.yml` erhöhen | +| `relation "xxx" does not exist` | API neu starten (Schema wird automatisch initialisiert) | +| `asyncpg.errors.InvalidCatalogName` | `POSTGRES_DB` in `.env` prüfen | +| CORS-Fehler im Browser | `ALLOWED_ORIGINS` in `main.py` anpassen | + +--- + +## 8. SSL/TLS für externen Zugriff + +### 8.1 Reverse-Proxy mit Caddy (empfohlen) + +Caddy automatisiert TLS-Zertifikate (Let's Encrypt). + +```caddyfile +# Caddyfile — /etc/caddy/Caddyfile + +# HTTPS mit automatischem TLS +nsct.example.com { + reverse_proxy nsct-api:8080 + + # Header-Sicherheit + header { + Strict-Transport-Security "max-age=31536000; includeSubDomains" + X-Content-Type-Options "nosniff" + X-Frame-Options "DENY" + } + + # Rate-Limiting + request_body { + size 10MB + } +} +``` + +**Caddy in Docker Compose:** + +```yaml +services: + caddy: + image: caddy:2-alpine + container_name: nsct-caddy + ports: + - "443:443" + - "80:80" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile + - caddy_data:/data + - caddy_config:/config + networks: + - nsct-internal + restart: unless-stopped +``` + +### 8.2 Reverse-Proxy mit Nginx + +```nginx +# /etc/nginx/sites-available/nsct +server { + listen 443 ssl http2; + server_name nsct.example.com; + + ssl_certificate /etc/letsencrypt/live/nsct.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/nsct.example.com/privkey.pem; + + # TLS-Konfiguration + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers HIGH:!aNULL:!MD5; + ssl_prefer_server_ciphers on; + ssl_session_cache shared:SSL:10m; + ssl_session_timeout 10m; + + # Security Headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-Frame-Options "DENY" always; + + # Rate Limiting + limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; + limit_req zone=api burst=20 nodelay; + + location / { + proxy_pass http://nsct-api:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Request-ID $request_id; + + # Timeouts für lange Research-Pipelines + proxy_read_timeout 300s; + proxy_connect_timeout 10s; + proxy_send_timeout 300s; + + # Body-Größe + client_max_body_size 10M; + + # Buffering deaktivieren für SSE/Streaming + proxy_buffering off; + } + + # Swagger UI nur whitelisted + location /docs { + allow 10.0.0.0/8; + deny all; + proxy_pass http://nsct-api:8080/docs; + } +} +``` + +### 8.3 HTTPS-Zertifikate + +**Mit certbot (Nginx):** + +```bash +# Certificate anfordern +sudo certbot --nginx -d nsct.example.com + +# Certificate renew test +sudo certbot renew --dry-run + +# Manuell renew +sudo certbot renew +``` + +**Mit Caddy (automatisch):** + +Caddy holt und renewt Zertifikate automatisch — keine manuelle Konfiguration nötig. + +### 8.4 DNS-Konfiguration + +```bash +# A-Record erstellen (IPv4) +# nsct.example.com. 300 IN A + +# AAAA-Record (IPv6, optional) +# nsct.example.com. 300 IN AAAA + +# Test DNS-Auflösung +dig nsct.example.com +nslookup nsct.example.com +``` + +--- + +## 9. Monitoring + +### 9.1 Prometheus-Metriken + +Der `GET /metrics` Endpunkt (Standard: aktiv) liefert Prometheus-kompatible Metriken: + +```bash +curl -s http://localhost:8080/metrics +``` + +**Metriken-Übersicht:** + +| Metrik | Typ | Beschreibung | +|--------|-----|-------------| +| `search_queries_total` | counter | Anzahl Suchanfragen | +| `sources_fetched_total` | counter | Anzahl abgerufener Quellen | +| `claims_extracted_total` | counter | Extrahierte Claims gesamt | +| `contradictions_detected_total` | counter | Festgestellte Widersprüche | +| `research_completed_total` | counter | Abgeschlossene Research-Läufe | +| `research_failed_total` | counter | Fehlgeschlagene Research-Läufe | +| `research_duration_seconds` | histogram | Dauer pro Research-Lauf | +| `llm_request_duration_seconds` | histogram | Dauer pro LLM-Anfrage | +| `active_research_runs` | gauge | Laufende Research-Läufe | + +**Steuerung über Environment:** + +```env +# Metriken-Endpoint aktivieren/deaktivieren +NSCT_METRICS_ENABLED=true # default +NSCT_METRICS_ENABLED=false # deaktivieren +``` + +### 9.2 Log-Ebenen + +```env +# Log-Level konfigurieren +NSCT_LOG_LEVEL=INFO # DEBUG | INFO | WARNING | ERROR | CRITICAL +``` + +**Log-Format (JSON):** + +```json +{ + "timestamp": "2025-09-06T12:00:00", + "level": "INFO", + "module": "nsct.api.main", + "message": "HTTP GET /health 200 2.34ms", + "request_id": "a1b2c3d4-...", + "method": "GET", + "path": "/health", + "status_code": "200", + "duration_ms": 2.34 +} +``` + +**Log-Ausgabeort:** + +```env +# Logs nach stdout (default) +NSCT_LOG_FILE= + +# Logs in Datei schreiben +NSCT_LOG_FILE=/app/data/nsct.log +``` + +### 9.3 Alerting-Empfehlungen + +```yaml +# Prometheus Alerting Rules (prometheus.yml) +# Beispiel: Research-Fehler-Rate alert +groups: + - name: nsct + rules: + - alert: HighResearchFailureRate + expr: rate(research_failed_total[5m]) > 0.1 + for: 2m + labels: + severity: warning + annotations: + summary: "Hohe Research-Fehler-Rate" + description: "Mehr als 10% Research-Läufe fehlschlagen." + + - alert: LLMUnreachable + expr: ready{status="not_ready"} == 1 + for: 1m + labels: + severity: critical + annotations: + summary: "LLM nicht erreichbar" + description: "Readiness-Check schlägt fehl." + + - alert: HighErrorRate + expr: rate(http_requests_total{status_code=~"5.."}[5m]) > 0.05 + for: 2m + labels: + severity: warning + annotations: + summary: "Hohe 5xx-Fehler-Rate" + description: "Mehr als 5% aller Requests enden mit 5xx." +``` + +**Prometheus-Target-Konfiguration:** + +```yaml +# scrape_configs +- job_name: 'nsct-api' + static_configs: + - targets: ['nsct-api:8080'] + metrics_path: '/metrics' + scrape_interval: 15s +``` + +### 9.4 Grafana-Dashboard (empfohlene Panels) + +| Panel | Metrik | Typ | +|-------|--------|-----| +| Research-Rate | `rate(research_completed_total[5m])` | Gauge | +| Fehlerrate | `rate(research_failed_total[5m])` | Gauge | +| LLM-Latenz | `histogram_quantile(0.95, research_duration_seconds_bucket)` | Histogram | +| Aktive Runs | `active_research_runs` | Gauge | +| Quellen pro Lauf | Quellen-Counter / Research-Counter | Ratio | +| DB-Größe | `pg_database_size('nsct')` | Gauge | + +--- + +## 10. Deployment-Checkliste + +### 10.1 Pre-Deploy-Checkliste + +- [ ] `.env` korrekt ausgefüllt und nicht in Git committet +- [ ] `POSTGRES_PASSWORD` stark genug (mind. 16 Zeichen, Sonderzeichen) +- [ ] `NSCT_LLM_API_KEY` gültig und zugreifbar +- [ ] `NSCT_LLM_BASE_URL` erreichbar (curl-Test vom Host aus) +- [ ] Resource Limits in `docker-compose.yml` an Hardware angepasst +- [ ] `NSCT_LOG_LEVEL` auf `INFO` für Production gesetzt +- [ ] `NSCT_DEBUG=false` (Swagger deaktiviert) +- [ ] Firewall-Regeln: nur Port 443 (HTTPS) offen, nicht 8080 +- [ ] Backup-Policy für PostgreSQL-Volume eingerichtet +- [ ] SSL/TLS-Zertifikat bereitgestellt (Caddy oder certbot) +- [ ] DNS-Einträge verifiziert (`dig nsct.example.com`) +- [ ] CORS-Origins an Production-URL angepasst +- [ ] `.dockerignore` enthält `.env`, `.venv`, `__pycache__` +- [ ] Git-Status sauber (keine uncommitted Änderungen) +- [ ] Reverse-Proxy konfiguriert und getestet +- [ ] Rate-Limiting aktiviert (Proxy-Ebene) + +### 10.2 Post-Deploy-Verifikation + +```bash +# 1. Container laufen? +docker compose ps +# Alle 3 Services sollten "healthy" sein + +# 2. Health-Check +curl -s http://localhost:8080/health | jq +# Erwartet: {"status": "ok", "version": "0.1.0"} + +# 3. Readiness-Check +curl -s http://localhost:8080/ready | jq +# Erwartet: {"status": "ready", "llm": "ok", ...} + +# 4. Provider-Status +curl -s http://localhost:8080/providers | jq +# Alle konfigurierten Provider sollten available=true zeigen + +# 5. Logs prüfen +docker compose logs --tail=20 nsct-api +# Keine ERROR-Level Einträge nach Startup + +# 6. Research-Test (vollständig) +curl -s http://localhost:8080/v1/research \ + -H "Content-Type: application/json" \ + -d '{ + "query": "Test recherche", + "language": "de", + "depth": "quick" + }' | jq + +# 7. Metriken prüfen +curl -s http://localhost:8080/metrics | head -20 + +# 8. Reverse-Proxy Test (HTTPS) +curl -s -I https://nsct.example.com/health | head -5 + +# 9. DB-Verbindung +docker compose exec postgres pg_isready -U nsct -d nsct + +# 10. Resource-Usage +docker stats --no-stream nsct-api nsct-postgres +``` + +### 10.3 Rollback-Prozedur + +```bash +# --- Rollback auf vorherige Version --- + +# 1. Vorheriges Image taggen (vor dem Update gemacht?) +# docker tag nsct-api nsct-api:rollback-$(date +%Y%m%d) + +# 2. Alte Version deployen +git checkout +docker compose down +docker compose up --build -d + +# 3. Datenbank-Backup vor Rollback? +# Falls Migrationen rückgängig gemacht werden müssen: +docker compose exec postgres pg_dump -U nsct -d nsct > pre_rollback_backup.sql + +# 4. Health-Check nach Rollback +sleep 20 +curl -s http://localhost:8080/health | jq +curl -s http://localhost:8080/ready | jq + +# 5. Logs auf Fehler prüfen +docker compose logs --tail=50 nsct-api | grep -i error + +# 6. Restore aus Backup (nur falls benötigt) +# gunzip -c nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct +``` + +### 10.4 Disaster Recovery + +```bash +# --- Komplette Wiederherstellung --- + +# 1. Neue Instanz aufsetzen (frisches System) +# Docker, Docker Compose, Git-Clone etc. + +# 2. .env aus Backup wiederherstellen +cp /backup/.env ./nsct/.env + +# 3. PostgreSQL-Datenbank wiederherstellen +gunzip -c /backup/nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct + +# 4. API starten +docker compose up -d nsct-api + +# 5. Verifizierung +curl -s http://localhost:8080/health | jq +curl -s http://localhost:8080/ready | jq +docker compose exec postgres psql -U nsct -d nsct -c "SELECT count(*) FROM research_runs;" +``` + +### 10.5 Wartungsplan + +```bash +# --- Regelmäßige Wartung --- + +# Monatlich: +# 1. PostgreSQL-Bereinigung (alte Research-Läufe) +docker compose exec postgres psql -U nsct -d nsct -c " +SELECT relname, pg_size_pretty(pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename))) +FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename)) DESC;" + +# 2. Log-Rotation prüfen +# journalctl --vacuum-time=7d (falls systemd-logs verwendet) + +# 3. Zertifikat-Ablauf prüfen +openssl x509 -in /etc/ssl/certs/nsct.pem -noout -enddate + +# 4. Docker-Prune (Platz freimachen) +docker system prune -f --filter "until=168h" # alte Images löschen +``` + +--- + +## Anhang: Schnelle Referenz + +### Wichtige Ports + +| Dienst | Port | Zugriff | +|--------|------|---------| +| NSCT API | 8080 | Intern (Docker) / 443 extern (Reverse-Proxy) | +| PostgreSQL | 5432 | Intern (Docker) | +| SearXNG | 8888 | Intern (Docker) | + +### Wichtige Pfade + +| Pfad | Beschreibung | +|------|-------------| +| `/.env` | Environment-Konfiguration | +| `/docker-compose.yml` | Container-Definition | +| `/Dockerfile` | Container-Build | +| `/src/nsct/api/` | API-Router | +| `/src/nsct/config.py` | Config-Module | +| `/src/nsct/logging_config.py` | Logging-Setup | +| `/src/nsct/metrics.py` | Prometheus-Metriken | +| `/src/nsct/security/policy.py` | SSRF-Schutz | + +### Kommando-Zusammenfassung + +```bash +# Deployment +docker compose up --build -d # Deploy +docker compose down # Stop +docker compose restart # Restart + +# Debugging +docker compose logs -f nsct-api # Logs +docker compose exec postgres psql ... # DB-Zugriff +docker compose exec nsct-api ... # API-Container-Shell + +# Health +curl localhost:8080/health # Liveness +curl localhost:8080/ready # Readiness +curl localhost:8080/metrics # Prometheus + +# Backup +docker compose exec postgres pg_dump -U nsct -d nsct > backup.sql +docker compose exec postgres pg_restore ... + +# Monitoring +docker stats # Resource Usage +docker compose ps # Container Status +``` \ No newline at end of file