NSCT---Neutral-Search-Crawler-Tool/ADMIN-Backend.md

# ADMIN-Handbook — NSCT Backend (API)

Produziert vom NSCT Development Team — Stand: 2025-09-06

---

## 1. Übersicht

**NSCT** (Neutral Search Crawler Tool) ist ein lokal betreibbares, containerisiertes Recherche- und Analyse-System. Es durchsucht automatisch Webquellen, extrahiert Behauptungen (Claims), bewertet Evidenz multi-dimensional und generiert neutrale, quellengestützte Berichte.

### Architektur-Überblick

```
┌─────────────────────────────────────────────────────────────────────┐
│                         Docker Compose Stack                       │
│                                                                      │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐                   │
│  │ nsct-api │────▶│ postgres │────▶│ searxng  │                   │
│  │ :8080    │     │ :5432    │     │ :8888    │                   │
│  │ FastAPI  │     │ PG 16    │     │ SearXNG  │                   │
│  └──────────┘     └──────────┘     └──────────┘                   │
│                                                                      │
│  User ───▶ LLM-Provider (:8030)  (extern, z.B. Ollama/VLLM)        │
│  User ───▶ Audio-Provider (:8030) (extern, STT/TTS)                │
│  User ───▶ Vision-Provider  (:8030) (extern, Bildanalyse)           │
└─────────────────────────────────────────────────────────────────────┘
```

**Kernprinzipien:**
- Container-first — alles läuft in Docker Compose
- Environment-only Configuration — keine Config-Dateien, alles über `.env`
- Security-first — SSRF-Schutz, non-root, read-only FS, dropped capabilities
- Provenance-Pflicht — jeder Schritt ist nachverfolgbar und reproduzierbar

**Module:**

| Modul | Path | Beschreibung |
|-------|------|-------------|
| API Layer | `src/nsct/api/` | REST-Endpunkte (FastAPI) |
| Models | `src/nsct/models/` | Pydantic v2 Schemas (DTOs) |
| Storage | `src/nsct/storage/` | SQLAlchemy 2.0 + asyncpg |
| Providers | `src/nsct/providers/` | LLM/Vision/Audio/Search Abstraktionen |
| Security | `src/nsct/security/` | SSRF-Schutz, URL-Validierung |
| Config | `src/nsct/config.py` | Environment-only, Pydantic BaseSettings |
| Logging | `src/nsct/logging_config.py` | JSON-Formatter, per-request context |
| Metrics | `src/nsct/metrics.py` | Prometheus-kompatible Metriken |
| Orchestration | `src/nsct/orchestration/` | State-Machine, Budget-Limits |
| Stages | `src/nsct/stages/` | Pipeline-Stages 5–13 |
| Provenance | `src/nsct/provenance.py` | Deterministische Hashes, Audit-Trail |

---

## 2. Schnellstart

### 2.1 Voraussetzungen

- Docker ≥ 24.0
- Docker Compose ≥ 2.23
- Mindestens 2 GB freier RAM
- Zugang zu einem OpenAI-kompatiblen LLM-Endpunkt (Port 8030)
- Git installiert

### 2.2 Repository klonen

```bash
git clone https://df918b20ee2da2c2f92f8dd9bdbdb42ee0c6f9da@git.frerkc.de/opencode/NSCT---Neutral-Search-Crawler-Tool.git
cd nsct
```

### 2.3 Environment-Datei erstellen und konfigurieren

```bash
cp .env.example .env
```

Trage die folgenden Werte in `.env` ein:

```env
# ---- LLM (Hauptmodell, verpflichtend) ----
NSCT_LLM_BASE_URL=http://<LLM_HOST>:8030/openai/v1
NSCT_LLM_MODEL=Qwen3.6-35B
NSCT_LLM_MAX_CONCURRENCY=3
NSCT_LLM_API_KEY=<dein-api-key>

# ---- Vision (Bildanalyse, optional) ----
NSCT_VISION_BASE_URL=http://<LLM_HOST>:8030/openai/visual/v1
NSCT_VISION_MODEL=Qwen2.5-VL-3B
NSCT_VISION_API_KEY=<vision-api-key>

# ---- Audio (STT/TTS, optional) ----
NSCT_AUDIO_BASE_URL=http://<LLM_HOST>:8030/hermes-audio
NSCT_AUDIO_MODEL=default
NSCT_AUDIO_API_KEY=<audio-api-key>

# ---- PostgreSQL ----
POSTGRES_USER=nsct
POSTGRES_PASSWORD=<starkes-passwort>
POSTGRES_DB=nsct
NSCT_DB_URL=postgresql+asyncpg://nsct:<passwort>@postgres:5432/nsct

# ---- SearXNG (optional, interne Suche) ----
NSCT_SEARXNG_BASE_URL=http://searxng:8080

# ---- General ----
NSCT_DEBUG=false
```

**Umgebungsvariablen-Referenz:**

| Variable | Typ | Default | Beschreibung |
|----------|-----|---------|-------------|
| `NSCT_LLM_BASE_URL` | string | `""` | OpenAI-kompatibler LLM-Endpunkt |
| `NSCT_LLM_MODEL` | string | `""` | Modell-Name |
| `NSCT_LLM_MAX_CONCURRENCY` | int | `3` | Max parallele LLM-Requests |
| `NSCT_LLM_API_KEY` | string | `""` | API-Key für LLM-Provider |
| `NSCT_VISION_BASE_URL` | string | `""` | Vision-Endpunkt |
| `NSCT_VISION_MODEL` | string | `""` | Vision-Modell-Name |
| `NSCT_VISION_API_KEY` | string | `""` | API-Key für Vision |
| `NSCT_AUDIO_BASE_URL` | string | `""` | Audio-Endpunkt (STT/TTS) |
| `NSCT_AUDIO_MODEL` | string | `default` | Audio-Modell |
| `NSCT_AUDIO_API_KEY` | string | `""` | API-Key für Audio |
| `POSTGRES_USER` | string | `nsct` | PostgreSQL-User |
| `POSTGRES_PASSWORD` | string | `nsct_secret` | PostgreSQL-Passwort |
| `POSTGRES_DB` | string | `nsct` | PostgreSQL-Name |
| `NSCT_DB_URL` | string | `""` | DB-Connection-String |
| `NSCT_SEARXNG_BASE_URL` | string | `""` | SearXNG-Instance-URL |
| `NSCT_DEBUG` | bool | `false` | Debug-Modus (Swagger aktiv) |
| `NSCT_LOG_LEVEL` | string | `INFO` | Log-Level (DEBUG/INFO/WARNING/ERROR/CRITICAL) |
| `NSCT_METRICS_ENABLED` | bool | `true` | `/metrics` Endpoint aktivieren |

### 2.4 Dienste starten

```bash
# Vollständiger Stack mit Build
docker compose up --build -d

# Oder nur API + PostgreSQL (ohne SearXNG)
docker compose up --build -d nsct-api postgres
```

### 2.5 Health-Check prüfen

```bash
# Grundlegender Liveness-Check (immer ok, wenn Prozess lebt)
curl -s http://localhost:8080/health | jq

# Readiness-Check (prüft LLM-Erreichbarkeit)
curl -s http://localhost:8080/ready | jq

# Provider-Status (zeigt konfigurierte Models)
curl -s http://localhost:8080/providers | jq
```

**Erwartete Antwort `/health`:**
```json
{
  "status": "ok",
  "version": "0.1.0"
}
```

**Erwartete Antwort `/ready` (bei voller Konfiguration):**
```json
{
  "status": "ready",
  "llm": "ok",
  "database": "unknown",
  "llm_model": "Qwen3.6-35B"
}
```

### 2.6 curl-Tests

```bash
# Provider-Status anzeigen
curl -s http://localhost:8080/providers | jq

# Prometheus-Metriken (falls aktiv)
curl -s http://localhost:8080/metrics

# Swagger UI (nur bei NSCT_DEBUG=true)
# http://localhost:8080/docs
# ReDoc: http://localhost:8080/redoc
```

---

## 3. PostgreSQL Setup

### 3.1 Verbindung einrichten

Die Standard-Verbindungsdaten aus `.env`:

```
Host:     postgres (intern im Docker-Netzwerk)
Port:     5432
Database: nsct
User:     nsct
Password: <POSTGRES_PASSWORD aus .env>
```

Shell-Zugriff auf die DB:

```bash
# In den Container verbinden
docker compose exec postgres psql -U nsct -d nsct

# Oder von der Host-Shell (wenn Port 5432 gemappt ist)
PGPASSWORD=<passwort> psql -h localhost -U nsct -d nsct -p 5432
```

### 3.2 Backup (pg_dump)

```bash
# Vollständiges Dump (SQL-Format)
docker compose exec postgres pg_dump -U nsct -d nsct > nsct_backup_$(date +%Y%m%d).sql

# Komprimiertes Dump
docker compose exec postgres pg_dump -U nsct -d nsct | gzip > nsct_backup_$(date +%Y%m%d).sql.gz

# Nur Struktur (ohne Daten)
docker compose exec postgres pg_dump -U nsct -d nsct --schema-only > nsct_schema.sql

# Nur Daten (ohne Struktur)
docker compose exec postgres pg_dump -U nsct -d nsct --data-only > nsct_data.sql

# Backup in ein eigenes Volume (für Offline-Storage)
docker compose exec postgres pg_dump -U nsct -d nsct > /tmp/nsct_backup.sql
docker cp nsct-postgres:/tmp/nsct_backup.sql ./nsct_backup.sql
```

### 3.3 Restore (pg_restore)

```bash
# Restore aus SQL-Datei (psql-Format)
docker compose exec -T postgres psql -U nsct -d nsct < nsct_backup.sql

# Restore aus komprimierter Datei
gunzip -c nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct

# Restore aus pg_dumpall (wenn mit pg_dumpall gesichert)
docker compose exec -T postgres psql -U nsct -d nsct < nsct_full_backup.sql
```

### 3.4 Migrationen

Die Datenbank wird aktuell **manuell** über SQLAlchemy-Alchemy-Reflection initialisiert. Migrationen werden über Alembic in zukünftigen Stages implementiert.

```bash
# Aktuelle Tabellenliste
docker compose exec postgres psql -U nsct -d nsct -c "\dt"

# Tabellenschema anzeigen
docker compose exec postgres psql -U nsct -d nsct -c "\d+ sources"

# Datenbankgröße
docker compose exec postgres psql -U nsct -d nsct -c "SELECT pg_database_size('nsct');"

# Tabellen-Größen (Byte)
docker compose exec postgres psql -U nsct -d nsct -c "\l+"
docker compose exec postgres psql -U nsct -d nsct -c "SELECT relname, pg_size_pretty(pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename))) FROM pg_tables WHERE schemaname = 'public';"
```

### 3.5 Troubleshooting

```bash
# PostgreSQL ist nicht bereit — Connection refused
# Lösung: Warte bis health check bestanden ist
docker compose ps postgres

# Prüfe ob Container läuft
docker compose logs postgres

# Manueller health-check
docker compose exec postgres pg_isready -U nsct -d nsct

# Wenn Password-Auth fehlschlägt:
# .env prüfen: POSTGRES_PASSWORD muss mit dem Wert übereinstimmen,
# den du in NSCT_DB_URL verwendest.

# Connection-String validieren:
echo "$NSCT_DB_URL" | python3 -c "import sys; print(sys.stdin.read())"

# PostgreSQL-Fehler in Container-Logs
docker compose logs --tail=50 postgres
```

**Häufige Fehler:**

| Fehler | Ursache | Lösung |
|--------|---------|--------|
| `FATAL: database "nsct" does not exist` | `POSTGRES_DB` in `.env` falsch gesetzt | Prüfe `POSTGRES_DB` und `NSCT_DB_URL` |
| `FATAL: password authentication failed` | Password mismatch | `.env` korrigieren, `docker compose down -v && up` |
| `could not connect to server: Connection refused` | PostgreSQL noch nicht ready | Warte, prüfe `docker compose ps` |
| `FATAL: too many connections` | Pool exhausted (default: 100) | `max_connections` erhöhen oder Poolgröße senken |
| `relation "xxx" does not exist` | Tabellen nicht initialisiert | Prüfe, ob API korrekt gestartet ist |

---

## 4. LLM-Endpunkte konfigurieren

Das NSCT-Backend erwartet OpenAI-kompatible Endpunkte (Standard: Ollama, vLLM, LMStudio).

### 4.1 Qwen3.6-35B (Hauptmodell)

Das primäre LLM für Claim-Extraktion, Synthese, Evidence-Scoring und alle LLM-gesteuerten Stages.

```env
NSCT_LLM_BASE_URL=http://<LLM_HOST>:8030/openai/v1
NSCT_LLM_MODEL=Qwen3.6-35B
NSCT_LLM_MAX_CONCURRENCY=3
NSCT_LLM_API_KEY=<dein-api-key>
```

**Endpoint-Validierung:**

```bash
# Modellliste abfragen
curl -s http://<LLM_HOST>:8030/openai/v1/models | jq

# Erwartete Antwort:
# {
#   "data": [
#     { "id": "Qwen3.6-35B", "object": "model", "owned_by": "qwen" }
#   ]
# }

# Chat-Completion testen
curl -s http://<LLM_HOST>:8030/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <dein-api-key>" \
  -d '{
    "model": "Qwen3.6-35B",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 10
  }' | jq
```

### 4.2 Qwen2.5-VL-3B (Vision)

Für Bild- und Dokumentenanalyse (Screenshots, PDFs, Charts).

```env
NSCT_VISION_BASE_URL=http://<LLM_HOST>:8030/openai/visual/v1
NSCT_VISION_MODEL=Qwen2.5-VL-3B
NSCT_VISION_API_KEY=<vision-api-key>
```

**Endpoint-Validierung:**

```bash
# Models-Endpoint
curl -s http://<LLM_HOST>:8030/openai/visual/v1/models | jq

# Chat mit Bild (Base64) testen
curl -s http://<LLM_HOST>:8030/openai/visual/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <vision-api-key>" \
  -d '{
    "model": "Qwen2.5-VL-3B",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
          {"type": "text", "text": "Was siehst du?"}
        ]
      }
    ]
  }' | jq
```

### 4.3 Audio-Modell (STT/TTS)

Für Audio-Transkription (Stage 11) und Text-to-Speech.

```env
NSCT_AUDIO_BASE_URL=http://<LLM_HOST>:8030/hermes-audio
NSCT_AUDIO_MODEL=default
NSCT_AUDIO_API_KEY=<audio-api-key>
```

**Endpoint-Validierung:**

```bash
# Test-Request (STT)
curl -s http://<LLM_HOST>:8030/hermes-audio/transcribe \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <audio-api-key>" \
  -d '{
    "model": "default",
    "language": "de"
  }' | jq
```

### 4.4 Endpoint-Validierung aus der API

```bash
# Alle Provider auf einmal prüfen
curl -s http://localhost:8080/providers | jq

# Readiness-Check (LLM-Erreichbarkeit)
curl -s http://localhost:8080/ready | jq
```

### 4.5 Troubleshooting

| Fehler | Ursache | Lösung |
|--------|---------|--------|
| `LLM connectivity check failed` | Endpunkt nicht erreichbar | Prüfe Netzwerk, Firewall, `.env` |
| `error:ConnectionRefused` | LLM-Server down | Starte LLM-Server neu |
| `error:401` | API-Key falsch | `.env` prüfen, Key regenerieren |
| `error:404` | Falscher Endpoint-Path | Prüfe Base-URL (z.B. `/openai/v1` vs `/v1`) |
| `timeout` | LLM zu langsam | `NSCT_LLM_MAX_CONCURRENCY` senken, Timeout erhöhen |

**LLM-Server-Status von innen prüfen:**

```bash
# Vom nsct-api Container aus zum LLM-Host ping'en
docker compose run --rm nsct-api curl -s http://<LLM_HOST>:8030/openai/v1/models | jq
```

---

## 5. API-Key-Verwaltung

> **Hinweis:** In der aktuellen Version (Stage 22+) ist die Authentifizierung noch nicht implementiert. Die Struktur ist vorbereitet für zukünftige API-Key-Authentifizierung.

### 5.1 User anlegen (vorbereitete SQL-Struktur)

Die Datenbank-Struktur für User/API-Key-Management:

```sql
-- User-Tabelle (vorbereitet für Stage 22+)
CREATE TABLE IF NOT EXISTS users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    username VARCHAR(255) UNIQUE NOT NULL,
    email VARCHAR(255) UNIQUE,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

-- API-Key-Tabelle
CREATE TABLE IF NOT EXISTS api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    key_hash VARCHAR(255) NOT NULL,
    name VARCHAR(255),
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT NOW(),
    expires_at TIMESTAMP,
    last_used_at TIMESTAMP
);

-- Indexes
CREATE INDEX idx_api_keys_key_hash ON api_keys(key_hash);
CREATE INDEX idx_api_keys_user_id ON api_keys(user_id);
CREATE INDEX idx_api_keys_is_active ON api_keys(is_active);
```

### 5.2 User und API-Key anlegen

```sql
-- User erstellen
INSERT INTO users (username, email, is_active)
VALUES ('admin', 'admin@example.com', TRUE)
ON CONFLICT (username) DO NOTHING;

-- API-Key generieren (Hash speichern)
-- 1. Generiere einen sicheren Key (von der Admin-Shell)
python3 -c "import secrets, hashlib; key = secrets.token_hex(32); print('KEY:', key); print('HASH:', hashlib.sha256(key.encode()).hexdigest())"

-- 2. Hash in der DB speichern (behalte den Original-Key sicher!)
INSERT INTO api_keys (user_id, key_hash, name, is_active)
SELECT u.id, '<key-hash>', 'admin-key', TRUE
FROM users u WHERE u.username = 'admin';
```

### 5.3 API-Key-Hashes verwalten

```sql
-- Alle aktiven Keys mit User
SELECT ak.name, ak.is_active, ak.created_at, ak.expires_at, u.username, u.email
FROM api_keys ak
JOIN users u ON ak.user_id = u.id
WHERE ak.is_active = TRUE
ORDER BY ak.created_at DESC;

-- Key-Verwendung (last_used_at)
UPDATE api_keys SET last_used_at = NOW() WHERE key_hash = '<key-hash>';

-- Ablaufende Keys prüfen
SELECT ak.name, ak.expires_at, u.username
FROM api_keys ak
JOIN users u ON ak.user_id = u.id
WHERE ak.expires_at < NOW() AND ak.is_active = TRUE;

-- Inaktive Keys deaktivieren
UPDATE api_keys SET is_active = FALSE WHERE is_active = FALSE;
```

### 5.4 Debugging von Auth-Problemen

```sql
-- API-Key existiert und ist aktiv?
SELECT * FROM api_keys WHERE key_hash = '<key-hash>' AND is_active = TRUE;

-- User existiert?
SELECT * FROM users WHERE username = '<username>';

-- Hash-Korrelation prüfen
SELECT ak.key_hash, u.username, ak.is_active
FROM api_keys ak
JOIN users u ON ak.user_id = u.id
WHERE ak.key_hash = '<key-hash>';
```

---

## 6. Docker Compose Konfiguration

### 6.1 docker-compose.yml erklärt

```yaml
# NSCT — docker-compose.yml
# Services: nsct-api, postgres, optional searxng
```

### 6.2 Services

| Service | Image | Port | Beschreibung |
|---------|-------|------|-------------|
| `nsct-api` | `build: .` | `8080:8080` | FastAPI-Server |
| `postgres` | `postgres:16-alpine` | `5432:5432` | PostgreSQL 16 |
| `searxng` | `searxng/searxng:latest` | `8888:8080` | Interne Suche (optional) |

### 6.3 Service-Details

#### nsct-api

```yaml
nsct-api:
  build:
    context: .
    dockerfile: Dockerfile
  container_name: nsct-api
  ports:
    - "8080:8080"
  environment:
    NSCT_DB_URL: postgresql+asyncpg://...@postgres:5432/nsct
    NSCT_LLM_MAX_CONCURRENCY: "3"
  depends_on:
    postgres:
      condition: service_healthy
    searxng:
      condition: service_started
  volumes:
    - nsct_data:/app/data
  healthcheck:
    test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8080/health')\" || exit 1"]
    interval: 30s
    timeout: 5s
    retries: 3
    start_period: 15s
  read_only: true
  tmpfs:
    - /tmp
  cap_drop:
    - ALL
```

**Umgebungsvariablen im Container:**
- `NSCT_DB_URL` — DB-URL (Host: `postgres`)
- `NSCT_LLM_MAX_CONCURRENCY` — LLM-Concurrency (erbt aus `.env`)
- Alle anderen vars aus `.env` (`env_file: .env`)

#### postgres

```yaml
postgres:
  image: postgres:16-alpine
  container_name: nsct-postgres
  environment:
    POSTGRES_USER: ${POSTGRES_USER:-nsct}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-nsct_secret}
    POSTGRES_DB: ${POSTGRES_DB:-nsct}
  ports:
    - "5432:5432"
  volumes:
    - postgres_data:/var/lib/postgresql/data
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-nsct} -d ${POSTGRES_DB:-nsct}"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 10s
  deploy:
    resources:
      limits:
        memory: 512M
        cpus: "0.5"
```

#### searxng (optional)

```yaml
searxng:
  image: searxng/searxng:latest
  container_name: nsct-searxng
  ports:
    - "8888:8080"
  volumes:
    - searxng_settings:/etc/searxng
  environment:
    - SEARXNG_BASE_URL=http://localhost:8888/
    - SEARXNG_SECRET_KEY=nsct_searxng_secret_key_change_me
  read_only: true
  tmpfs:
    - /tmp
  cap_drop:
    - ALL
```

### 6.4 Volumes

| Volume | Pfad | Beschreibung |
|--------|------|-------------|
| `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL-Datenbankdateien |
| `nsct_data` | `/app/data` | NSCT-Runtime-Daten (Cache, etc.) |
| `searxng_settings` | `/etc/searxng` | SearXNG-Konfiguration |

### 6.5 Networks

Standard `docker compose` Netzwerk (bridge). Alle Services sind über Service-Namen erreichbar:

```
postgres     → postgres:5432
searxng      → searxng:8080
nsct-api     → nsct-api:8080
```

Für isolierte Netzwerke:

```yaml
networks:
  nsct-internal:
    driver: bridge
```

### 6.6 Resource Limits

| Service | Memory Limit | CPU Limit | Memory Reservation |
|---------|-------------|-----------|-------------------|
| nsct-api | 1G | 1.0 | 256M |
| postgres | 512M | 0.5 | — |
| searxng | 512M | 0.5 | — |

**Anpassen:** In `docker-compose.yml` unter `deploy.resources.limits` editieren.

### 6.7 Health Checks

| Service | Endpoint | Interval | Timeout | Retries |
|---------|----------|----------|---------|---------|
| nsct-api | `GET /health` | 30s | 5s | 3 |
| postgres | `pg_isready` | 10s | 5s | 5 |

---

## 7. Health-Check, Logs, Troubleshooting

### 7.1 curl-Befehle

```bash
# Basic health — Liveness
curl -s http://localhost:8080/health | jq

# Readiness — prüft LLM-Connectivity
curl -s http://localhost:8080/ready | jq

# Provider-Info
curl -s http://localhost:8080/providers | jq

# Prometheus-Metriken
curl -s http://localhost:8080/metrics

# Swagger Docs (nur bei NSCT_DEBUG=true)
curl -s -I http://localhost:8080/docs
```

### 7.2 Docker Logs

```bash
# Alle Logs (live)
docker compose logs -f

# Nur API-Container
docker compose logs -f nsct-api

# Letzte 100 Zeilen
docker compose logs --tail=100 nsct-api

# Errors nur
docker compose logs -f nsct-api | grep ERROR

# PostgreSQL Logs
docker compose logs -f postgres

# Logs mit Zeitstempeln
docker compose logs -f --timestamp nsct-api
```

### 7.3 Container-Status prüfen

```bash
# Container-Status
docker compose ps

# Container-Details (JSON)
docker inspect nsct-api | jq '.[0].State'

# Container-Logs (letzte 50 Zeilen, nicht-live)
docker compose logs --tail=50 nsct-api

# Resource-Verbrauch
docker stats nsct-api nsct-postgres nsct-searxng
```

### 7.4 Restart-Prozedur

```bash
# Soft restart (alle Services)
docker compose restart

# Nur API neu starten
docker compose restart nsct-api

# Vollständiger Reload (neuer Build)
docker compose down
docker compose up --build -d

# Container komplett entfernen und neu erstellen
docker compose down -v
docker compose up --build -d
```

### 7.5 Common Errors und Lösungen

| Error | Lösung |
|-------|--------|
| `Connection refused` auf `/health` | Warte 15s (start_period), prüfe `docker compose ps` |
| `LLM: error:Connection refused` | LLM-Server prüfen, `.env` Base-URL validieren |
| `LLM: error:ConnectionRefused` | Netzwerk-Verbindung prüfen: `docker compose run --rm nsct-api curl -v <LLM_URL>` |
| `HTTP 503` bei Research-Requests | Readiness prüfen: `curl localhost:8080/ready` |
| `no space left on device` | `docker system prune -f` ausführen |
| Container OOMKilled | Memory-Limit in `docker-compose.yml` erhöhen |
| `relation "xxx" does not exist` | API neu starten (Schema wird automatisch initialisiert) |
| `asyncpg.errors.InvalidCatalogName` | `POSTGRES_DB` in `.env` prüfen |
| CORS-Fehler im Browser | `ALLOWED_ORIGINS` in `main.py` anpassen |

---

## 8. SSL/TLS für externen Zugriff

### 8.1 Reverse-Proxy mit Caddy (empfohlen)

Caddy automatisiert TLS-Zertifikate (Let's Encrypt).

```caddyfile
# Caddyfile — /etc/caddy/Caddyfile

# HTTPS mit automatischem TLS
nsct.example.com {
    reverse_proxy nsct-api:8080

    # Header-Sicherheit
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
    }

    # Rate-Limiting
    request_body {
        size 10MB
    }
}
```

**Caddy in Docker Compose:**

```yaml
services:
  caddy:
    image: caddy:2-alpine
    container_name: nsct-caddy
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - nsct-internal
    restart: unless-stopped
```

### 8.2 Reverse-Proxy mit Nginx

```nginx
# /etc/nginx/sites-available/nsct
server {
    listen 443 ssl http2;
    server_name nsct.example.com;

    ssl_certificate     /etc/letsencrypt/live/nsct.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/nsct.example.com/privkey.pem;

    # TLS-Konfiguration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    # Security Headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "DENY" always;

    # Rate Limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req zone=api burst=20 nodelay;

    location / {
        proxy_pass http://nsct-api:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-ID $request_id;

        # Timeouts für lange Research-Pipelines
        proxy_read_timeout 300s;
        proxy_connect_timeout 10s;
        proxy_send_timeout 300s;

        # Body-Größe
        client_max_body_size 10M;

        # Buffering deaktivieren für SSE/Streaming
        proxy_buffering off;
    }

    # Swagger UI nur whitelisted
    location /docs {
        allow 10.0.0.0/8;
        deny all;
        proxy_pass http://nsct-api:8080/docs;
    }
}
```

### 8.3 HTTPS-Zertifikate

**Mit certbot (Nginx):**

```bash
# Certificate anfordern
sudo certbot --nginx -d nsct.example.com

# Certificate renew test
sudo certbot renew --dry-run

# Manuell renew
sudo certbot renew
```

**Mit Caddy (automatisch):**

Caddy holt und renewt Zertifikate automatisch — keine manuelle Konfiguration nötig.

### 8.4 DNS-Konfiguration

```bash
# A-Record erstellen (IPv4)
# nsct.example.com. 300 IN A  <SERVER_IP>

# AAAA-Record (IPv6, optional)
# nsct.example.com. 300 IN AAAA  <SERVER_IPV6>

# Test DNS-Auflösung
dig nsct.example.com
nslookup nsct.example.com
```

---

## 9. Monitoring

### 9.1 Prometheus-Metriken

Der `GET /metrics` Endpunkt (Standard: aktiv) liefert Prometheus-kompatible Metriken:

```bash
curl -s http://localhost:8080/metrics
```

**Metriken-Übersicht:**

| Metrik | Typ | Beschreibung |
|--------|-----|-------------|
| `search_queries_total` | counter | Anzahl Suchanfragen |
| `sources_fetched_total` | counter | Anzahl abgerufener Quellen |
| `claims_extracted_total` | counter | Extrahierte Claims gesamt |
| `contradictions_detected_total` | counter | Festgestellte Widersprüche |
| `research_completed_total` | counter | Abgeschlossene Research-Läufe |
| `research_failed_total` | counter | Fehlgeschlagene Research-Läufe |
| `research_duration_seconds` | histogram | Dauer pro Research-Lauf |
| `llm_request_duration_seconds` | histogram | Dauer pro LLM-Anfrage |
| `active_research_runs` | gauge | Laufende Research-Läufe |

**Steuerung über Environment:**

```env
# Metriken-Endpoint aktivieren/deaktivieren
NSCT_METRICS_ENABLED=true   # default
NSCT_METRICS_ENABLED=false  # deaktivieren
```

### 9.2 Log-Ebenen

```env
# Log-Level konfigurieren
NSCT_LOG_LEVEL=INFO    # DEBUG | INFO | WARNING | ERROR | CRITICAL
```

**Log-Format (JSON):**

```json
{
  "timestamp": "2025-09-06T12:00:00",
  "level": "INFO",
  "module": "nsct.api.main",
  "message": "HTTP GET /health 200 2.34ms",
  "request_id": "a1b2c3d4-...",
  "method": "GET",
  "path": "/health",
  "status_code": "200",
  "duration_ms": 2.34
}
```

**Log-Ausgabeort:**

```env
# Logs nach stdout (default)
NSCT_LOG_FILE=

# Logs in Datei schreiben
NSCT_LOG_FILE=/app/data/nsct.log
```

### 9.3 Alerting-Empfehlungen

```yaml
# Prometheus Alerting Rules (prometheus.yml)
# Beispiel: Research-Fehler-Rate alert
groups:
  - name: nsct
    rules:
      - alert: HighResearchFailureRate
        expr: rate(research_failed_total[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Hohe Research-Fehler-Rate"
          description: "Mehr als 10% Research-Läufe fehlschlagen."

      - alert: LLMUnreachable
        expr: ready{status="not_ready"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "LLM nicht erreichbar"
          description: "Readiness-Check schlägt fehl."

      - alert: HighErrorRate
        expr: rate(http_requests_total{status_code=~"5.."}[5m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Hohe 5xx-Fehler-Rate"
          description: "Mehr als 5% aller Requests enden mit 5xx."
```

**Prometheus-Target-Konfiguration:**

```yaml
# scrape_configs
- job_name: 'nsct-api'
  static_configs:
    - targets: ['nsct-api:8080']
  metrics_path: '/metrics'
  scrape_interval: 15s
```

### 9.4 Grafana-Dashboard (empfohlene Panels)

| Panel | Metrik | Typ |
|-------|--------|-----|
| Research-Rate | `rate(research_completed_total[5m])` | Gauge |
| Fehlerrate | `rate(research_failed_total[5m])` | Gauge |
| LLM-Latenz | `histogram_quantile(0.95, research_duration_seconds_bucket)` | Histogram |
| Aktive Runs | `active_research_runs` | Gauge |
| Quellen pro Lauf | Quellen-Counter / Research-Counter | Ratio |
| DB-Größe | `pg_database_size('nsct')` | Gauge |

---

## 10. Deployment-Checkliste

### 10.1 Pre-Deploy-Checkliste

- [ ] `.env` korrekt ausgefüllt und nicht in Git committet
- [ ] `POSTGRES_PASSWORD` stark genug (mind. 16 Zeichen, Sonderzeichen)
- [ ] `NSCT_LLM_API_KEY` gültig und zugreifbar
- [ ] `NSCT_LLM_BASE_URL` erreichbar (curl-Test vom Host aus)
- [ ] Resource Limits in `docker-compose.yml` an Hardware angepasst
- [ ] `NSCT_LOG_LEVEL` auf `INFO` für Production gesetzt
- [ ] `NSCT_DEBUG=false` (Swagger deaktiviert)
- [ ] Firewall-Regeln: nur Port 443 (HTTPS) offen, nicht 8080
- [ ] Backup-Policy für PostgreSQL-Volume eingerichtet
- [ ] SSL/TLS-Zertifikat bereitgestellt (Caddy oder certbot)
- [ ] DNS-Einträge verifiziert (`dig nsct.example.com`)
- [ ] CORS-Origins an Production-URL angepasst
- [ ] `.dockerignore` enthält `.env`, `.venv`, `__pycache__`
- [ ] Git-Status sauber (keine uncommitted Änderungen)
- [ ] Reverse-Proxy konfiguriert und getestet
- [ ] Rate-Limiting aktiviert (Proxy-Ebene)

### 10.2 Post-Deploy-Verifikation

```bash
# 1. Container laufen?
docker compose ps
# Alle 3 Services sollten "healthy" sein

# 2. Health-Check
curl -s http://localhost:8080/health | jq
# Erwartet: {"status": "ok", "version": "0.1.0"}

# 3. Readiness-Check
curl -s http://localhost:8080/ready | jq
# Erwartet: {"status": "ready", "llm": "ok", ...}

# 4. Provider-Status
curl -s http://localhost:8080/providers | jq
# Alle konfigurierten Provider sollten available=true zeigen

# 5. Logs prüfen
docker compose logs --tail=20 nsct-api
# Keine ERROR-Level Einträge nach Startup

# 6. Research-Test (vollständig)
curl -s http://localhost:8080/v1/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Test recherche",
    "language": "de",
    "depth": "quick"
  }' | jq

# 7. Metriken prüfen
curl -s http://localhost:8080/metrics | head -20

# 8. Reverse-Proxy Test (HTTPS)
curl -s -I https://nsct.example.com/health | head -5

# 9. DB-Verbindung
docker compose exec postgres pg_isready -U nsct -d nsct

# 10. Resource-Usage
docker stats --no-stream nsct-api nsct-postgres
```

### 10.3 Rollback-Prozedur

```bash
# --- Rollback auf vorherige Version ---

# 1. Vorheriges Image taggen (vor dem Update gemacht?)
# docker tag nsct-api nsct-api:rollback-$(date +%Y%m%d)

# 2. Alte Version deployen
git checkout <commit-hash>
docker compose down
docker compose up --build -d

# 3. Datenbank-Backup vor Rollback?
# Falls Migrationen rückgängig gemacht werden müssen:
docker compose exec postgres pg_dump -U nsct -d nsct > pre_rollback_backup.sql

# 4. Health-Check nach Rollback
sleep 20
curl -s http://localhost:8080/health | jq
curl -s http://localhost:8080/ready | jq

# 5. Logs auf Fehler prüfen
docker compose logs --tail=50 nsct-api | grep -i error

# 6. Restore aus Backup (nur falls benötigt)
# gunzip -c nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct
```

### 10.4 Disaster Recovery

```bash
# --- Komplette Wiederherstellung ---

# 1. Neue Instanz aufsetzen (frisches System)
# Docker, Docker Compose, Git-Clone etc.

# 2. .env aus Backup wiederherstellen
cp /backup/.env ./nsct/.env

# 3. PostgreSQL-Datenbank wiederherstellen
gunzip -c /backup/nsct_backup.sql.gz | docker compose exec -T postgres psql -U nsct -d nsct

# 4. API starten
docker compose up -d nsct-api

# 5. Verifizierung
curl -s http://localhost:8080/health | jq
curl -s http://localhost:8080/ready | jq
docker compose exec postgres psql -U nsct -d nsct -c "SELECT count(*) FROM research_runs;"
```

### 10.5 Wartungsplan

```bash
# --- Regelmäßige Wartung ---

# Monatlich:
# 1. PostgreSQL-Bereinigung (alte Research-Läufe)
docker compose exec postgres psql -U nsct -d nsct -c "
SELECT relname, pg_size_pretty(pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename)))
FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(quote_ident(schemaname) || '.' || quote_ident(tablename)) DESC;"

# 2. Log-Rotation prüfen
# journalctl --vacuum-time=7d  (falls systemd-logs verwendet)

# 3. Zertifikat-Ablauf prüfen
openssl x509 -in /etc/ssl/certs/nsct.pem -noout -enddate

# 4. Docker-Prune (Platz freimachen)
docker system prune -f --filter "until=168h"  # alte Images löschen
```

---

## Anhang: Schnelle Referenz

### Wichtige Ports

| Dienst | Port | Zugriff |
|--------|------|---------|
| NSCT API | 8080 | Intern (Docker) / 443 extern (Reverse-Proxy) |
| PostgreSQL | 5432 | Intern (Docker) |
| SearXNG | 8888 | Intern (Docker) |

### Wichtige Pfade

| Pfad | Beschreibung |
|------|-------------|
| `/.env` | Environment-Konfiguration |
| `/docker-compose.yml` | Container-Definition |
| `/Dockerfile` | Container-Build |
| `/src/nsct/api/` | API-Router |
| `/src/nsct/config.py` | Config-Module |
| `/src/nsct/logging_config.py` | Logging-Setup |
| `/src/nsct/metrics.py` | Prometheus-Metriken |
| `/src/nsct/security/policy.py` | SSRF-Schutz |

### Kommando-Zusammenfassung

```bash
# Deployment
docker compose up --build -d          # Deploy
docker compose down                    # Stop
docker compose restart                 # Restart

# Debugging
docker compose logs -f nsct-api        # Logs
docker compose exec postgres psql ...  # DB-Zugriff
docker compose exec nsct-api ...       # API-Container-Shell

# Health
curl localhost:8080/health             # Liveness
curl localhost:8080/ready              # Readiness
curl localhost:8080/metrics            # Prometheus

# Backup
docker compose exec postgres pg_dump -U nsct -d nsct > backup.sql
docker compose exec postgres pg_restore ...

# Monitoring
docker stats                           # Resource Usage
docker compose ps                      # Container Status
```