opencode/NSCT---Neutral-Search-Crawler-Tool
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
50 Commits 1 Branch 0 Tags
9c83f305b07fbfdadcf51eb223803bfef3cf4959
Commit Graph

50 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
faligam
9c83f305b0 Document dynamic research route fix 2026-09-06 17:45:19 +02:00
faligam
7c41f9bdde Document browser session and API contract fixes 2026-09-06 17:39:44 +02:00
faligam
b0aaddd236 Document browser flow error banner fix 2026-09-06 17:29:53 +02:00
faligam
fd1250ce2e Update handoff for deployed API-key testing 
Record the backend and frontend authentication commits, the rebuilt local stacks, and the next browser-to-backend end-to-end test procedure.
 2026-09-06 17:25:09 +02:00
faligam
1aacf4aa20 Implement persistent API-key authentication 
Protect the research lifecycle with X-API-Key validation backed by persistent user and key records. Store only salted scrypt hashes, support expiry and revocation, and expose a local admin CLI for create/list/revoke workflows.

Initialize only the authentication schema at startup, prevent SQL echo from exposing sensitive bound values, and keep health probes public. Add coverage for valid, missing, invalid, expired, and revoked keys.

Document deployment and key administration, update the local CLI to send NSCT_API_KEY, and record the reset handoff state.
 2026-09-06 17:23:05 +02:00
faligam
64eb4e8465 Update reset handoff for API authentication work 2026-09-06 16:59:34 +02:00
faligam
9830acf446 Document remote frontend integration 2026-09-06 16:55:48 +02:00
faligam
fddb4da1c7 Update ADMIN-Backend.md 2026-09-06 14:24:51 +00:00
faligam
ac905e7198 Fix Docker installation startup 2026-09-06 13:01:01 +02:00
NSCT Agent
8a918c9d86 Fix: Replace GitHub URL with git.frerkc.de in Admin Handbook 2026-09-06 09:52:26 +00:00
NSCT Agent
8276975e93 Admin Handbook: Backend (NSCT API) 2026-09-06 08:37:06 +00:00
NSCT Agent
9ae1a7ba21 stage22: Abschluss & Production Readiness — E2E-Tests, CHANGELOG, Dokumentation 2026-09-05 15:03:48 +00:00
NSCT Agent
8ea6269f9a feat(stage21): implement reproducibility — research_run_hash, full provenance tracking 
- compute_research_run_hash(): deterministic SHA256 from query+budget+created_at
- ProvenanceEntry / ProvenanceLog: step-by-step trace with inputs, outputs, metadata, errors
- ResearchRun model: research_run_hash field (64-char hex)
- ResearchRunProvenance SQLAlchemy table for DB persistence
- Orchestrator: auto-log provenance before/after each pipeline step
- 35 tests: hash determinism, entry/log methods, storage model, integration
 2026-09-05 14:27:45 +00:00
NSCT Agent
f9b761ced7 fix: priority_queue async context methods (async_fix for Python 3.14+). Add test_priority_limiter.py tests for async context manager 2026-09-05 13:36:49 +00:00
NSCT Agent
6e2e7386ad stage20: context budgeting - per-stage token limits (planner 12k, claim 16k, contradiction 24k, synthesis 48k) 
Implemented:
- context_budget.py: ContextBudgetConfig (Pydantic, frozen) mit 4 Stage-Limits,
  validation (ge/le), get_limit(), total_max_tokens, stage_keys
- context_budget.py: ContextBudgetTracker mit track_tokens(), get_usage(),
  is_exhausted(), reset_stage(), reset_all(), total_usage, elapsed_seconds
- context_budget.py: ContextBudgetExhaustedError mit stage_name, used_tokens, limit_tokens
- orchestrator.py: _track_context_tokens() Methode, context_budget_config/tracker init
- orchestrator.py: context_budget_tracker property export
- __init__.py: exports ContextBudgetConfig, ContextBudgetExhaustedError, ContextBudgetTracker
- priority_queue.py: __aenter__/__aexit__ auf async geandert (Testfix)
- test_context_budget.py: 28 Tests (DefaultConfig, TrackAccumulation, ExhaustedError,
  GetUsage, Reset, IsExhausted, InvalidStageNames, InvalidTokens)
- test_context_budget_integration.py: 6 Tests (Orchestrator-Integration)
- HANDOFF.md: Stage 20 abgeschlossen dokumentiert

Tests: 34 passed (28 unit + 6 integration) + 14 performance = 48 total
 2026-08-29 08:57:18 +00:00
NSCT Agent
58488de7b3 docs: update HANDOFF.md — Stage 19 completed, 20 next 2026-08-28 17:32:32 +00:00
NSCT Agent
4335a40d68 stage19: performance optimierung — LLM concurrency semaphore, priority (HIGH/NORMAL/LOW), limiter 2026-08-28 17:28:31 +00:00
NSCT Agent
ce9b031ea0 docs: update HANDOFF.md — Stage 18 completed, 19 next 2026-08-28 16:23:43 +00:00
NSCT Agent
60679083fa stage18: docker hardening — non-root user, read-only FS, no-new-privileges, health-check fix, .dockerignore 2026-08-28 16:22:23 +00:00
NSCT Agent
b23ff5a565 docs: update HANDOFF.md — Stages 14-16 completed, 17 next 2026-08-27 20:35:40 +00:00
NSCT Agent
413050bdd9 docs: update HANDOFF.md — Stage 16 completed, Stage 17 next 2026-08-27 19:52:04 +00:00
NSCT Agent
08ec0e290c feat(stage16): implement observability — structured logging, metrics, tracing 
- JSONFormatter, ContextVars, set_research_run_id() with short logger names
- self-built Counter/Histogram/Gauge system (no external deps)
- Prometheus text export at /metrics
- Request logging middleware with X-Request-ID
- Metrics instrumentation: search_queries, sources_fetched, claims, contradictions
- Histograms: research_duration, llm_request_duration
- Gauge: active_research_runs
- 13 + 21 = 34 tests
 2026-08-27 19:44:46 +00:00
NSCT Agent
b2340fca63 docs: update HANDOFF.md — Stage 15 completed, Stage 16 next 2026-08-27 17:11:29 +00:00
NSCT Agent
1cc17cf9ac feat(stage15): implement CLI with search, status, report, sources, claims, list, delete 
- argparse subcommands: help, search, status, report, sources, claims, list, delete
- httpx client with NSCT_API_URL env var (default http://localhost:8080)
- ANSI colored output: green/completed, red/failed, yellow/running
- search command with --depth (quick/normal/deep) and --language (de/en)
- Polling loop for search with Ctrl+C interrupt
- ASCII table formatting for sources/claims/list
- Connection error handling with clear messages
- 28 tests for all commands and helper functions
 2026-08-27 17:09:36 +00:00
NSCT Agent
c059fb0ffd docs: update HANDOFF.md — Stage 14 completed, Stage 15 next 2026-08-27 16:14:51 +00:00
NSCT Agent
4d5f6747f7 feat(stage14): implement REST API for research lifecycle 
- POST /v1/research — start research (non-blocking, background pipeline)
- GET  /v1/research/{id} — research metadata
- GET  /v1/research/{id}/status — detailed state machine status
- GET  /v1/research/{id}/sources — sources list
- GET  /v1/research/{id}/claims — claims list
- GET  /v1/research/{id}/evidence — evidence scores
- GET  /v1/research/{id}/report — research report
- DELETE /v1/research/{id} — delete research (non-completed)
- GET  /v1/research — paginated list of all research runs

Depth budgets (quick/normal/deep) control only resource limits.
In-memory store for now, to be replaced with PostgreSQL later.
Router mounted in main.py as tag "research-api".
 2026-08-27 16:07:47 +00:00
Frerk Campen
a55e26f3b1 docs: update HANDOFF.md — Stage 13 completed, Stage 14 next 2026-08-26 11:22:02 +00:00
Frerk Campen
0b7a624bc8 feat(stage13): implement Iterative Research / Gap Analysis\n\nImplement Gap Analysis Engine (Stage 13):\n- GapAnalysisEngine: detect single-source claims, contradictions,\n missing primary sources, weak evidence\n- IterationReport: structured gap findings with severity & target\n- GapSearchQuery: derived search queries per gap finding\n- Integration into ResearchOrchestrator: runs gap analysis after\n extracting, then executes gap searches iteratively\n- 17 tests covering all analysis categories and edge cases 2026-08-26 11:19:35 +00:00
NSCT Agent
2ef7b67002 feat(stage12): implement Research Orchestrator 2026-08-26 10:29:57 +00:00
NSCT Agent
825aedb057 feat(stage11): implement audio integration 2026-08-25 18:37:31 +00:00
NSCT Agent
4b8ae6a41a fix(stage11): resolve subagent merge conflicts — audio models, vision fix, test fixes 2026-08-25 17:29:17 +00:00
NSCT Agent
d76d48a0ee docs: update HANDOFF.md — Stages 10+11 completed, Stage 12 next 2026-08-25 15:42:00 +00:00
NSCT Agent
10a083aa02 feat(stage11): audio integration — STT for interviews, podcasts, press conferences with timestamped claims 2026-08-25 15:31:03 +00:00
NSCT Agent
3ab875d2bc feat(stage10): implement vision integration 2026-08-25 14:42:29 +00:00
NSCT Agent
14f016457a tests(stage9): fix test_stage9_synthesis.py 2026-08-25 14:08:15 +00:00
NSCT Agent
8582a8e60f docs: update HANDOFF.md — Stage 9 completed, Stage 10 next 2026-08-24 12:38:47 +00:00
NSCT Agent
d87e2b4d14 feat(stage9): neutral synthesis engine — LLM-generated report from evidence package 2026-08-24 11:52:10 +00:00
NSCT Agent
1b54b172ca docs: update HANDOFF.md — Stage 8 completed, Stage 9 next 2026-08-24 07:52:43 +00:00
NSCT Agent
719e218d9a feat(stage8): evidence scoring — transparent multidimensional scores (independence, proximity, support, contradiction, directness, date) 2026-08-24 07:47:00 +00:00
NSCT Agent
cc9d4f2ba5 docs: update HANDOFF.md — Stage 7 completed, Stage 8 next 2026-08-23 21:00:40 +00:00
NSCT Agent
d1e6bb6cf4 feat(stage7): claim clustering & contradiction candidates — semantic grouping, numeric normalization, pairwise analysis 
- ClaimClusterModel: LLM-basierte semantische Gruppierung von Claims
- ClaimRelationModel: SUPPORTS, CONTRADICTS, DUPLICATE, UNCERTAIN pairwise relations
- ClaimNLUModel: numerische Normalisierung (%, Währungen, deutsche/englische Wörter)
- stage7_normalize_numerics.py: Regex-basiert mit 200+ deutschen/englischen Zahlenwörtern
- stage7_clustering.py: LLM-Clustering + pairwise claim-relation analysis
- API: POST cluster-claims, GET clusters, GET claim-relations
- 79 tests: numerische Normalisierung, LLM-Parsing, Clustering, Relationen, Edge-Cases
- Dedup: claims mit gleichen numerischen Werten werden zusammengefasst
 2026-08-23 20:57:17 +00:00
NSCT Agent
d2294c58d0 docs: update HANDOFF.md — Stage 6 completed, Stage 7 next 2026-08-23 18:50:25 +00:00
NSCT Agent
a60cf21a2c feat(stage6): source independence & citation graph — detect syndication, shared origins, text similarity 
- SourceIndependenceModel: per-source independence_score (0.0-1.0), syndication_group_id, primary_source_id, content_hash, shared_urls
- CitationGraphEdgeModel: directed edges (SYNDICATED, QUOTES, LINKS_TO, REPOST, SIMILAR_CONTENT) with confidence + evidence
- Content-Hash (SHA-256): instant syndication detection for identical content
- difflib Vorfilterung: >60% → LLM, >80% → high confidence, 100% → immediate syndication
- LLM-Pairwise-Analysis: two-text-comparison for suspicious pairs only (bounded concurrency)
- independence_score: 1.0 base, -0.4 for syndicated, -0.1 per high-similarity pair
- Pydantic schemas: SourceIndependenceScore, CitationGraphEdge, SyntacticSimilarityResult, LlmSyndicationAnalysis, SourceIndependenceAnalysisResult
- LLM response parser: handles JSON, markdown code blocks, partial/invalid JSON
- 43 tests: content hash, similarity thresholds, LLM parsing, analyzer integration, edge cases, prompt templates
 2026-08-23 18:47:24 +00:00
NSCT Agent
27e494d161 docs: update HANDOFF.md — Stage 5 completed, Stage 6 next 2026-08-23 17:58:41 +00:00
NSCT Agent
e8b6515f67 feat(stage5): implement claim extraction — atomic verifiable claims from sources 
- Claim model with provenance, evidence_span, attribution, claim_type
- Stage5Extractor: LLM-based atomic claim extraction from source content
  - Never summarizes — always extracts atomic, verifiable claims
  - Claims require evidence span (exact quote from source)
  - Attribution per claim (who says what)
  - Claim types: fact, opinion, prediction, recommendation, claim
  - Confidence score 0.0–1.0 per claim
  - Bounded concurrency, SSRF-safe, max content truncation
- REST API: GET/POST /research/{run_id}/claims
- 36 tests: parsing, edge cases, integration, validation
 2026-08-23 17:56:01 +00:00
NSCT Agent
b8181deb05 Stage 4: Research Planner — LLM-basierte Recherchestrategie-Generierung 
- src/nsct/agents/planner.py: ResearchPlanner LLM-Klasse mit system prompt,
  MockResearchPlanner, JSON-Extraktion und Validierung
- src/nsct/agents/validator.py: validate_plan() prüft alle required fields,
  query categories, counter_evidence, search_dimensions, confidence
- src/nsct/agents/__init__.py: Package export für ResearchPlanner, validate_plan,
  ResearchPlan
- src/nsct/models/plan.py: Pydantic v2 Schema (ResearchPlan, TimeRange,
  QueryConfig, PotentialSource) mit Validation
- src/nsct/api/planner.py: POST /research/planner Endpoint mit Debug-Support
- src/nsct/api/main.py: Mount des planner routers
- src/nsct/crawler/pdf.py: exportiere extract_pdf_content als Alias
- src/nsct/api/crawler.py: Pydantic BaseModel für Request-Models
- tests/test_planner.py: 25 Tests für Planner, Validator, Schema, API

- Search-Bias-Reduktion: 6+ Query-Typen, counter_evidence, beide Seiten
- Keine TODOs, keine unvollständigen Funktionen
- Alle Dateien syntaktisch korrekt und getestet
 2026-08-23 13:28:23 +00:00
NSCT Agent
a8595cc950 Stage 3: Crawler und Content Extraction 
- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
  (50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
  content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
  Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
  localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
  /crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
  Error Handling, Content Hash Determinismus
 2026-08-23 12:53:19 +00:00
NSCT Agent
a1ef260520 STAGE 2: Search Provider Abstraction für NSCT 
- SearchProvider-Interface mit abstract.base, NormalizedResult-Modell
- DuckDuckGoProvider: HTTP-basierte Suche ohne API-Keys, Fallback-fähig
- MultiProviderSearch: parallele Suche, URL-Dedup, Provider-Config, Fallback
- POST /search-Endpoint mit normalisierten Ergebnissen, Debug-Mode
- 25 unit tests: NormalizedResult, MultiProviderSearch, DuckDuckGoProvider
- rank ist KEIN truth_score - Dokumentation und Validierung durchgängig
 2026-08-23 12:16:29 +00:00
NSCT Agent
9280d69ebf Stage 1: OpenAI-compatible provider layer (llm, vision, audio, metrics, debug) 2026-08-23 11:54:50 +00:00
NSCT Agent
e9410be941 Stage 0: Repository und Architekturgrundlage 
- Pyproject.toml mit FastAPI, Pydantic v2, SQLAlchemy, httpx, asyncio,
  BeautifulSoup4, selectolax, trafilatura, uvicorn, pytest-asyncio
- Multi-stage Dockerfile (Python 3.12-slim, Non-Root-User nsct)
- docker-compose.yml (nsct-api + postgres + optional searxng)
- .env.example mit allen Config-Parametern
- Config-System: AppSettings mit LLMConfig, VisionConfig, AudioConfig,
  DatabaseConfig — komplett aus Environment, keine Hardcodes
- Strukturiertes Logging mit research_id/llm_request_id Tracking
- Pydantic v2 Schemas: SearchQuery, Source, Claim, EvidenceRelation,
  CitationEdge, ResearchReport
- SQLAlchemy 2.0 Declarative Models + async Engine Factory
- SSRF-Schutz: URL-Validation, IP-Blocklist (RFC1918, Cloud Metadata,
  file://, ftp://)
- Provider-Interfaces: LLMProvider, VisionProvider, AudioProvider,
  SearchProvider, ContentFetcher als ABCs
- Health-Endpoints: /health, /ready (LLM-Connect-Test), /providers
- FastAPI App mit CORS, lifespan (LLM Pre-Flight)
- CLI-Stub mit Entry-Points: nsct, nsct-core, nsct-api
- 6 Test-Cases: /health, /ready, /providers + No-Secrets-Test
- Vollständige Dokumentation: README, ARCHITECTURE, SECURITY,
  METHODOLOGY, API, DEPLOYMENT
- .gitignore (Python, Docker, IDE, .env)
 2026-08-23 11:33:45 +00:00