opencode/NSCT---Neutral-Search-Crawler-Tool
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
56 Commits 1 Branch 0 Tags
a5324d39711e683817c7dd0afe7ce96ec01409e5
Commit Graph

28 Commits

Author SHA1 Message Date
faligam
a5324d3971 Fix research pipeline runtime failures 2026-09-07 11:16:36 +02:00
faligam
e5b7d90436 Fix research pipeline provider failures 2026-09-06 18:32:42 +02:00
faligam
1aacf4aa20 Implement persistent API-key authentication 
Protect the research lifecycle with X-API-Key validation backed by persistent user and key records. Store only salted scrypt hashes, support expiry and revocation, and expose a local admin CLI for create/list/revoke workflows.

Initialize only the authentication schema at startup, prevent SQL echo from exposing sensitive bound values, and keep health probes public. Add coverage for valid, missing, invalid, expired, and revoked keys.

Document deployment and key administration, update the local CLI to send NSCT_API_KEY, and record the reset handoff state.
 2026-09-06 17:23:05 +02:00
NSCT Agent
9ae1a7ba21 stage22: Abschluss & Production Readiness — E2E-Tests, CHANGELOG, Dokumentation 2026-09-05 15:03:48 +00:00
NSCT Agent
8ea6269f9a feat(stage21): implement reproducibility — research_run_hash, full provenance tracking 
- compute_research_run_hash(): deterministic SHA256 from query+budget+created_at
- ProvenanceEntry / ProvenanceLog: step-by-step trace with inputs, outputs, metadata, errors
- ResearchRun model: research_run_hash field (64-char hex)
- ResearchRunProvenance SQLAlchemy table for DB persistence
- Orchestrator: auto-log provenance before/after each pipeline step
- 35 tests: hash determinism, entry/log methods, storage model, integration
 2026-09-05 14:27:45 +00:00
NSCT Agent
f9b761ced7 fix: priority_queue async context methods (async_fix for Python 3.14+). Add test_priority_limiter.py tests for async context manager 2026-09-05 13:36:49 +00:00
NSCT Agent
6e2e7386ad stage20: context budgeting - per-stage token limits (planner 12k, claim 16k, contradiction 24k, synthesis 48k) 
Implemented:
- context_budget.py: ContextBudgetConfig (Pydantic, frozen) mit 4 Stage-Limits,
  validation (ge/le), get_limit(), total_max_tokens, stage_keys
- context_budget.py: ContextBudgetTracker mit track_tokens(), get_usage(),
  is_exhausted(), reset_stage(), reset_all(), total_usage, elapsed_seconds
- context_budget.py: ContextBudgetExhaustedError mit stage_name, used_tokens, limit_tokens
- orchestrator.py: _track_context_tokens() Methode, context_budget_config/tracker init
- orchestrator.py: context_budget_tracker property export
- __init__.py: exports ContextBudgetConfig, ContextBudgetExhaustedError, ContextBudgetTracker
- priority_queue.py: __aenter__/__aexit__ auf async geandert (Testfix)
- test_context_budget.py: 28 Tests (DefaultConfig, TrackAccumulation, ExhaustedError,
  GetUsage, Reset, IsExhausted, InvalidStageNames, InvalidTokens)
- test_context_budget_integration.py: 6 Tests (Orchestrator-Integration)
- HANDOFF.md: Stage 20 abgeschlossen dokumentiert

Tests: 34 passed (28 unit + 6 integration) + 14 performance = 48 total
 2026-08-29 08:57:18 +00:00
NSCT Agent
4335a40d68 stage19: performance optimierung — LLM concurrency semaphore, priority (HIGH/NORMAL/LOW), limiter 2026-08-28 17:28:31 +00:00
NSCT Agent
60679083fa stage18: docker hardening — non-root user, read-only FS, no-new-privileges, health-check fix, .dockerignore 2026-08-28 16:22:23 +00:00
NSCT Agent
08ec0e290c feat(stage16): implement observability — structured logging, metrics, tracing 
- JSONFormatter, ContextVars, set_research_run_id() with short logger names
- self-built Counter/Histogram/Gauge system (no external deps)
- Prometheus text export at /metrics
- Request logging middleware with X-Request-ID
- Metrics instrumentation: search_queries, sources_fetched, claims, contradictions
- Histograms: research_duration, llm_request_duration
- Gauge: active_research_runs
- 13 + 21 = 34 tests
 2026-08-27 19:44:46 +00:00
NSCT Agent
1cc17cf9ac feat(stage15): implement CLI with search, status, report, sources, claims, list, delete 
- argparse subcommands: help, search, status, report, sources, claims, list, delete
- httpx client with NSCT_API_URL env var (default http://localhost:8080)
- ANSI colored output: green/completed, red/failed, yellow/running
- search command with --depth (quick/normal/deep) and --language (de/en)
- Polling loop for search with Ctrl+C interrupt
- ASCII table formatting for sources/claims/list
- Connection error handling with clear messages
- 28 tests for all commands and helper functions
 2026-08-27 17:09:36 +00:00
NSCT Agent
4d5f6747f7 feat(stage14): implement REST API for research lifecycle 
- POST /v1/research — start research (non-blocking, background pipeline)
- GET  /v1/research/{id} — research metadata
- GET  /v1/research/{id}/status — detailed state machine status
- GET  /v1/research/{id}/sources — sources list
- GET  /v1/research/{id}/claims — claims list
- GET  /v1/research/{id}/evidence — evidence scores
- GET  /v1/research/{id}/report — research report
- DELETE /v1/research/{id} — delete research (non-completed)
- GET  /v1/research — paginated list of all research runs

Depth budgets (quick/normal/deep) control only resource limits.
In-memory store for now, to be replaced with PostgreSQL later.
Router mounted in main.py as tag "research-api".
 2026-08-27 16:07:47 +00:00
Frerk Campen
0b7a624bc8 feat(stage13): implement Iterative Research / Gap Analysis\n\nImplement Gap Analysis Engine (Stage 13):\n- GapAnalysisEngine: detect single-source claims, contradictions,\n missing primary sources, weak evidence\n- IterationReport: structured gap findings with severity & target\n- GapSearchQuery: derived search queries per gap finding\n- Integration into ResearchOrchestrator: runs gap analysis after\n extracting, then executes gap searches iteratively\n- 17 tests covering all analysis categories and edge cases 2026-08-26 11:19:35 +00:00
NSCT Agent
825aedb057 feat(stage11): implement audio integration 2026-08-25 18:37:31 +00:00
NSCT Agent
4b8ae6a41a fix(stage11): resolve subagent merge conflicts — audio models, vision fix, test fixes 2026-08-25 17:29:17 +00:00
NSCT Agent
10a083aa02 feat(stage11): audio integration — STT for interviews, podcasts, press conferences with timestamped claims 2026-08-25 15:31:03 +00:00
NSCT Agent
3ab875d2bc feat(stage10): implement vision integration 2026-08-25 14:42:29 +00:00
NSCT Agent
14f016457a tests(stage9): fix test_stage9_synthesis.py 2026-08-25 14:08:15 +00:00
NSCT Agent
8582a8e60f docs: update HANDOFF.md — Stage 9 completed, Stage 10 next 2026-08-24 12:38:47 +00:00
NSCT Agent
d87e2b4d14 feat(stage9): neutral synthesis engine — LLM-generated report from evidence package 2026-08-24 11:52:10 +00:00
NSCT Agent
719e218d9a feat(stage8): evidence scoring — transparent multidimensional scores (independence, proximity, support, contradiction, directness, date) 2026-08-24 07:47:00 +00:00
NSCT Agent
d1e6bb6cf4 feat(stage7): claim clustering & contradiction candidates — semantic grouping, numeric normalization, pairwise analysis 
- ClaimClusterModel: LLM-basierte semantische Gruppierung von Claims
- ClaimRelationModel: SUPPORTS, CONTRADICTS, DUPLICATE, UNCERTAIN pairwise relations
- ClaimNLUModel: numerische Normalisierung (%, Währungen, deutsche/englische Wörter)
- stage7_normalize_numerics.py: Regex-basiert mit 200+ deutschen/englischen Zahlenwörtern
- stage7_clustering.py: LLM-Clustering + pairwise claim-relation analysis
- API: POST cluster-claims, GET clusters, GET claim-relations
- 79 tests: numerische Normalisierung, LLM-Parsing, Clustering, Relationen, Edge-Cases
- Dedup: claims mit gleichen numerischen Werten werden zusammengefasst
 2026-08-23 20:57:17 +00:00
NSCT Agent
a60cf21a2c feat(stage6): source independence & citation graph — detect syndication, shared origins, text similarity 
- SourceIndependenceModel: per-source independence_score (0.0-1.0), syndication_group_id, primary_source_id, content_hash, shared_urls
- CitationGraphEdgeModel: directed edges (SYNDICATED, QUOTES, LINKS_TO, REPOST, SIMILAR_CONTENT) with confidence + evidence
- Content-Hash (SHA-256): instant syndication detection for identical content
- difflib Vorfilterung: >60% → LLM, >80% → high confidence, 100% → immediate syndication
- LLM-Pairwise-Analysis: two-text-comparison for suspicious pairs only (bounded concurrency)
- independence_score: 1.0 base, -0.4 for syndicated, -0.1 per high-similarity pair
- Pydantic schemas: SourceIndependenceScore, CitationGraphEdge, SyntacticSimilarityResult, LlmSyndicationAnalysis, SourceIndependenceAnalysisResult
- LLM response parser: handles JSON, markdown code blocks, partial/invalid JSON
- 43 tests: content hash, similarity thresholds, LLM parsing, analyzer integration, edge cases, prompt templates
 2026-08-23 18:47:24 +00:00
NSCT Agent
e8b6515f67 feat(stage5): implement claim extraction — atomic verifiable claims from sources 
- Claim model with provenance, evidence_span, attribution, claim_type
- Stage5Extractor: LLM-based atomic claim extraction from source content
  - Never summarizes — always extracts atomic, verifiable claims
  - Claims require evidence span (exact quote from source)
  - Attribution per claim (who says what)
  - Claim types: fact, opinion, prediction, recommendation, claim
  - Confidence score 0.0–1.0 per claim
  - Bounded concurrency, SSRF-safe, max content truncation
- REST API: GET/POST /research/{run_id}/claims
- 36 tests: parsing, edge cases, integration, validation
 2026-08-23 17:56:01 +00:00
NSCT Agent
b8181deb05 Stage 4: Research Planner — LLM-basierte Recherchestrategie-Generierung 
- src/nsct/agents/planner.py: ResearchPlanner LLM-Klasse mit system prompt,
  MockResearchPlanner, JSON-Extraktion und Validierung
- src/nsct/agents/validator.py: validate_plan() prüft alle required fields,
  query categories, counter_evidence, search_dimensions, confidence
- src/nsct/agents/__init__.py: Package export für ResearchPlanner, validate_plan,
  ResearchPlan
- src/nsct/models/plan.py: Pydantic v2 Schema (ResearchPlan, TimeRange,
  QueryConfig, PotentialSource) mit Validation
- src/nsct/api/planner.py: POST /research/planner Endpoint mit Debug-Support
- src/nsct/api/main.py: Mount des planner routers
- src/nsct/crawler/pdf.py: exportiere extract_pdf_content als Alias
- src/nsct/api/crawler.py: Pydantic BaseModel für Request-Models
- tests/test_planner.py: 25 Tests für Planner, Validator, Schema, API

- Search-Bias-Reduktion: 6+ Query-Typen, counter_evidence, beide Seiten
- Keine TODOs, keine unvollständigen Funktionen
- Alle Dateien syntaktisch korrekt und getestet
 2026-08-23 13:28:23 +00:00
NSCT Agent
a8595cc950 Stage 3: Crawler und Content Extraction 
- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
  (50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
  content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
  Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
  localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
  /crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
  Error Handling, Content Hash Determinismus
 2026-08-23 12:53:19 +00:00
NSCT Agent
a1ef260520 STAGE 2: Search Provider Abstraction für NSCT 
- SearchProvider-Interface mit abstract.base, NormalizedResult-Modell
- DuckDuckGoProvider: HTTP-basierte Suche ohne API-Keys, Fallback-fähig
- MultiProviderSearch: parallele Suche, URL-Dedup, Provider-Config, Fallback
- POST /search-Endpoint mit normalisierten Ergebnissen, Debug-Mode
- 25 unit tests: NormalizedResult, MultiProviderSearch, DuckDuckGoProvider
- rank ist KEIN truth_score - Dokumentation und Validierung durchgängig
 2026-08-23 12:16:29 +00:00
NSCT Agent
e9410be941 Stage 0: Repository und Architekturgrundlage 
- Pyproject.toml mit FastAPI, Pydantic v2, SQLAlchemy, httpx, asyncio,
  BeautifulSoup4, selectolax, trafilatura, uvicorn, pytest-asyncio
- Multi-stage Dockerfile (Python 3.12-slim, Non-Root-User nsct)
- docker-compose.yml (nsct-api + postgres + optional searxng)
- .env.example mit allen Config-Parametern
- Config-System: AppSettings mit LLMConfig, VisionConfig, AudioConfig,
  DatabaseConfig — komplett aus Environment, keine Hardcodes
- Strukturiertes Logging mit research_id/llm_request_id Tracking
- Pydantic v2 Schemas: SearchQuery, Source, Claim, EvidenceRelation,
  CitationEdge, ResearchReport
- SQLAlchemy 2.0 Declarative Models + async Engine Factory
- SSRF-Schutz: URL-Validation, IP-Blocklist (RFC1918, Cloud Metadata,
  file://, ftp://)
- Provider-Interfaces: LLMProvider, VisionProvider, AudioProvider,
  SearchProvider, ContentFetcher als ABCs
- Health-Endpoints: /health, /ready (LLM-Connect-Test), /providers
- FastAPI App mit CORS, lifespan (LLM Pre-Flight)
- CLI-Stub mit Entry-Points: nsct, nsct-core, nsct-api
- 6 Test-Cases: /health, /ready, /providers + No-Secrets-Test
- Vollständige Dokumentation: README, ARCHITECTURE, SECURITY,
  METHODOLOGY, API, DEPLOYMENT
- .gitignore (Python, Docker, IDE, .env)
 2026-08-23 11:33:45 +00:00