NSCT Agent
d2294c58d0
docs: update HANDOFF.md — Stage 6 completed, Stage 7 next
2026-08-23 18:50:25 +00:00
NSCT Agent
a60cf21a2c
feat(stage6): source independence & citation graph — detect syndication, shared origins, text similarity
...
- SourceIndependenceModel: per-source independence_score (0.0-1.0), syndication_group_id, primary_source_id, content_hash, shared_urls
- CitationGraphEdgeModel: directed edges (SYNDICATED, QUOTES, LINKS_TO, REPOST, SIMILAR_CONTENT) with confidence + evidence
- Content-Hash (SHA-256): instant syndication detection for identical content
- difflib Vorfilterung: >60% → LLM, >80% → high confidence, 100% → immediate syndication
- LLM-Pairwise-Analysis: two-text-comparison for suspicious pairs only (bounded concurrency)
- independence_score: 1.0 base, -0.4 for syndicated, -0.1 per high-similarity pair
- Pydantic schemas: SourceIndependenceScore, CitationGraphEdge, SyntacticSimilarityResult, LlmSyndicationAnalysis, SourceIndependenceAnalysisResult
- LLM response parser: handles JSON, markdown code blocks, partial/invalid JSON
- 43 tests: content hash, similarity thresholds, LLM parsing, analyzer integration, edge cases, prompt templates
2026-08-23 18:47:24 +00:00
NSCT Agent
27e494d161
docs: update HANDOFF.md — Stage 5 completed, Stage 6 next
2026-08-23 17:58:41 +00:00
NSCT Agent
e8b6515f67
feat(stage5): implement claim extraction — atomic verifiable claims from sources
...
- Claim model with provenance, evidence_span, attribution, claim_type
- Stage5Extractor: LLM-based atomic claim extraction from source content
- Never summarizes — always extracts atomic, verifiable claims
- Claims require evidence span (exact quote from source)
- Attribution per claim (who says what)
- Claim types: fact, opinion, prediction, recommendation, claim
- Confidence score 0.0–1.0 per claim
- Bounded concurrency, SSRF-safe, max content truncation
- REST API: GET/POST /research/{run_id}/claims
- 36 tests: parsing, edge cases, integration, validation
2026-08-23 17:56:01 +00:00
NSCT Agent
b8181deb05
Stage 4: Research Planner — LLM-basierte Recherchestrategie-Generierung
...
- src/nsct/agents/planner.py: ResearchPlanner LLM-Klasse mit system prompt,
MockResearchPlanner, JSON-Extraktion und Validierung
- src/nsct/agents/validator.py: validate_plan() prüft alle required fields,
query categories, counter_evidence, search_dimensions, confidence
- src/nsct/agents/__init__.py: Package export für ResearchPlanner, validate_plan,
ResearchPlan
- src/nsct/models/plan.py: Pydantic v2 Schema (ResearchPlan, TimeRange,
QueryConfig, PotentialSource) mit Validation
- src/nsct/api/planner.py: POST /research/planner Endpoint mit Debug-Support
- src/nsct/api/main.py: Mount des planner routers
- src/nsct/crawler/pdf.py: exportiere extract_pdf_content als Alias
- src/nsct/api/crawler.py: Pydantic BaseModel für Request-Models
- tests/test_planner.py: 25 Tests für Planner, Validator, Schema, API
- Search-Bias-Reduktion: 6+ Query-Typen, counter_evidence, beide Seiten
- Keine TODOs, keine unvollständigen Funktionen
- Alle Dateien syntaktisch korrekt und getestet
2026-08-23 13:28:23 +00:00
NSCT Agent
a8595cc950
Stage 3: Crawler und Content Extraction
...
- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
(50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
/crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
Error Handling, Content Hash Determinismus
2026-08-23 12:53:19 +00:00
NSCT Agent
a1ef260520
STAGE 2: Search Provider Abstraction für NSCT
...
- SearchProvider-Interface mit abstract.base, NormalizedResult-Modell
- DuckDuckGoProvider: HTTP-basierte Suche ohne API-Keys, Fallback-fähig
- MultiProviderSearch: parallele Suche, URL-Dedup, Provider-Config, Fallback
- POST /search-Endpoint mit normalisierten Ergebnissen, Debug-Mode
- 25 unit tests: NormalizedResult, MultiProviderSearch, DuckDuckGoProvider
- rank ist KEIN truth_score - Dokumentation und Validierung durchgängig
2026-08-23 12:16:29 +00:00
NSCT Agent
9280d69ebf
Stage 1: OpenAI-compatible provider layer (llm, vision, audio, metrics, debug)
2026-08-23 11:54:50 +00:00
NSCT Agent
e9410be941
Stage 0: Repository und Architekturgrundlage
...
- Pyproject.toml mit FastAPI, Pydantic v2, SQLAlchemy, httpx, asyncio,
BeautifulSoup4, selectolax, trafilatura, uvicorn, pytest-asyncio
- Multi-stage Dockerfile (Python 3.12-slim, Non-Root-User nsct)
- docker-compose.yml (nsct-api + postgres + optional searxng)
- .env.example mit allen Config-Parametern
- Config-System: AppSettings mit LLMConfig, VisionConfig, AudioConfig,
DatabaseConfig — komplett aus Environment, keine Hardcodes
- Strukturiertes Logging mit research_id/llm_request_id Tracking
- Pydantic v2 Schemas: SearchQuery, Source, Claim, EvidenceRelation,
CitationEdge, ResearchReport
- SQLAlchemy 2.0 Declarative Models + async Engine Factory
- SSRF-Schutz: URL-Validation, IP-Blocklist (RFC1918, Cloud Metadata,
file://, ftp://)
- Provider-Interfaces: LLMProvider, VisionProvider, AudioProvider,
SearchProvider, ContentFetcher als ABCs
- Health-Endpoints: /health, /ready (LLM-Connect-Test), /providers
- FastAPI App mit CORS, lifespan (LLM Pre-Flight)
- CLI-Stub mit Entry-Points: nsct, nsct-core, nsct-api
- 6 Test-Cases: /health, /ready, /providers + No-Secrets-Test
- Vollständige Dokumentation: README, ARCHITECTURE, SECURITY,
METHODOLOGY, API, DEPLOYMENT
- .gitignore (Python, Docker, IDE, .env)
2026-08-23 11:33:45 +00:00
