opencode/NSCT---Neutral-Search-Crawler-Tool
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
34 Commits 1 Branch 0 Tags
4335a40d68ffb89c4ddddfd83ed5680e9820adac
Commit Graph

34 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
NSCT Agent
4335a40d68 stage19: performance optimierung — LLM concurrency semaphore, priority (HIGH/NORMAL/LOW), limiter 2026-08-28 17:28:31 +00:00
NSCT Agent
ce9b031ea0 docs: update HANDOFF.md — Stage 18 completed, 19 next 2026-08-28 16:23:43 +00:00
NSCT Agent
60679083fa stage18: docker hardening — non-root user, read-only FS, no-new-privileges, health-check fix, .dockerignore 2026-08-28 16:22:23 +00:00
NSCT Agent
b23ff5a565 docs: update HANDOFF.md — Stages 14-16 completed, 17 next 2026-08-27 20:35:40 +00:00
NSCT Agent
413050bdd9 docs: update HANDOFF.md — Stage 16 completed, Stage 17 next 2026-08-27 19:52:04 +00:00
NSCT Agent
08ec0e290c feat(stage16): implement observability — structured logging, metrics, tracing 
- JSONFormatter, ContextVars, set_research_run_id() with short logger names
- self-built Counter/Histogram/Gauge system (no external deps)
- Prometheus text export at /metrics
- Request logging middleware with X-Request-ID
- Metrics instrumentation: search_queries, sources_fetched, claims, contradictions
- Histograms: research_duration, llm_request_duration
- Gauge: active_research_runs
- 13 + 21 = 34 tests
 2026-08-27 19:44:46 +00:00
NSCT Agent
b2340fca63 docs: update HANDOFF.md — Stage 15 completed, Stage 16 next 2026-08-27 17:11:29 +00:00
NSCT Agent
1cc17cf9ac feat(stage15): implement CLI with search, status, report, sources, claims, list, delete 
- argparse subcommands: help, search, status, report, sources, claims, list, delete
- httpx client with NSCT_API_URL env var (default http://localhost:8080)
- ANSI colored output: green/completed, red/failed, yellow/running
- search command with --depth (quick/normal/deep) and --language (de/en)
- Polling loop for search with Ctrl+C interrupt
- ASCII table formatting for sources/claims/list
- Connection error handling with clear messages
- 28 tests for all commands and helper functions
 2026-08-27 17:09:36 +00:00
NSCT Agent
c059fb0ffd docs: update HANDOFF.md — Stage 14 completed, Stage 15 next 2026-08-27 16:14:51 +00:00
NSCT Agent
4d5f6747f7 feat(stage14): implement REST API for research lifecycle 
- POST /v1/research — start research (non-blocking, background pipeline)
- GET  /v1/research/{id} — research metadata
- GET  /v1/research/{id}/status — detailed state machine status
- GET  /v1/research/{id}/sources — sources list
- GET  /v1/research/{id}/claims — claims list
- GET  /v1/research/{id}/evidence — evidence scores
- GET  /v1/research/{id}/report — research report
- DELETE /v1/research/{id} — delete research (non-completed)
- GET  /v1/research — paginated list of all research runs

Depth budgets (quick/normal/deep) control only resource limits.
In-memory store for now, to be replaced with PostgreSQL later.
Router mounted in main.py as tag "research-api".
 2026-08-27 16:07:47 +00:00
Frerk Campen
a55e26f3b1 docs: update HANDOFF.md — Stage 13 completed, Stage 14 next 2026-08-26 11:22:02 +00:00
Frerk Campen
0b7a624bc8 feat(stage13): implement Iterative Research / Gap Analysis\n\nImplement Gap Analysis Engine (Stage 13):\n- GapAnalysisEngine: detect single-source claims, contradictions,\n missing primary sources, weak evidence\n- IterationReport: structured gap findings with severity & target\n- GapSearchQuery: derived search queries per gap finding\n- Integration into ResearchOrchestrator: runs gap analysis after\n extracting, then executes gap searches iteratively\n- 17 tests covering all analysis categories and edge cases 2026-08-26 11:19:35 +00:00
NSCT Agent
2ef7b67002 feat(stage12): implement Research Orchestrator 2026-08-26 10:29:57 +00:00
NSCT Agent
825aedb057 feat(stage11): implement audio integration 2026-08-25 18:37:31 +00:00
NSCT Agent
4b8ae6a41a fix(stage11): resolve subagent merge conflicts — audio models, vision fix, test fixes 2026-08-25 17:29:17 +00:00
NSCT Agent
d76d48a0ee docs: update HANDOFF.md — Stages 10+11 completed, Stage 12 next 2026-08-25 15:42:00 +00:00
NSCT Agent
10a083aa02 feat(stage11): audio integration — STT for interviews, podcasts, press conferences with timestamped claims 2026-08-25 15:31:03 +00:00
NSCT Agent
3ab875d2bc feat(stage10): implement vision integration 2026-08-25 14:42:29 +00:00
NSCT Agent
14f016457a tests(stage9): fix test_stage9_synthesis.py 2026-08-25 14:08:15 +00:00
NSCT Agent
8582a8e60f docs: update HANDOFF.md — Stage 9 completed, Stage 10 next 2026-08-24 12:38:47 +00:00
NSCT Agent
d87e2b4d14 feat(stage9): neutral synthesis engine — LLM-generated report from evidence package 2026-08-24 11:52:10 +00:00
NSCT Agent
1b54b172ca docs: update HANDOFF.md — Stage 8 completed, Stage 9 next 2026-08-24 07:52:43 +00:00
NSCT Agent
719e218d9a feat(stage8): evidence scoring — transparent multidimensional scores (independence, proximity, support, contradiction, directness, date) 2026-08-24 07:47:00 +00:00
NSCT Agent
cc9d4f2ba5 docs: update HANDOFF.md — Stage 7 completed, Stage 8 next 2026-08-23 21:00:40 +00:00
NSCT Agent
d1e6bb6cf4 feat(stage7): claim clustering & contradiction candidates — semantic grouping, numeric normalization, pairwise analysis 
- ClaimClusterModel: LLM-basierte semantische Gruppierung von Claims
- ClaimRelationModel: SUPPORTS, CONTRADICTS, DUPLICATE, UNCERTAIN pairwise relations
- ClaimNLUModel: numerische Normalisierung (%, Währungen, deutsche/englische Wörter)
- stage7_normalize_numerics.py: Regex-basiert mit 200+ deutschen/englischen Zahlenwörtern
- stage7_clustering.py: LLM-Clustering + pairwise claim-relation analysis
- API: POST cluster-claims, GET clusters, GET claim-relations
- 79 tests: numerische Normalisierung, LLM-Parsing, Clustering, Relationen, Edge-Cases
- Dedup: claims mit gleichen numerischen Werten werden zusammengefasst
 2026-08-23 20:57:17 +00:00
NSCT Agent
d2294c58d0 docs: update HANDOFF.md — Stage 6 completed, Stage 7 next 2026-08-23 18:50:25 +00:00
NSCT Agent
a60cf21a2c feat(stage6): source independence & citation graph — detect syndication, shared origins, text similarity 
- SourceIndependenceModel: per-source independence_score (0.0-1.0), syndication_group_id, primary_source_id, content_hash, shared_urls
- CitationGraphEdgeModel: directed edges (SYNDICATED, QUOTES, LINKS_TO, REPOST, SIMILAR_CONTENT) with confidence + evidence
- Content-Hash (SHA-256): instant syndication detection for identical content
- difflib Vorfilterung: >60% → LLM, >80% → high confidence, 100% → immediate syndication
- LLM-Pairwise-Analysis: two-text-comparison for suspicious pairs only (bounded concurrency)
- independence_score: 1.0 base, -0.4 for syndicated, -0.1 per high-similarity pair
- Pydantic schemas: SourceIndependenceScore, CitationGraphEdge, SyntacticSimilarityResult, LlmSyndicationAnalysis, SourceIndependenceAnalysisResult
- LLM response parser: handles JSON, markdown code blocks, partial/invalid JSON
- 43 tests: content hash, similarity thresholds, LLM parsing, analyzer integration, edge cases, prompt templates
 2026-08-23 18:47:24 +00:00
NSCT Agent
27e494d161 docs: update HANDOFF.md — Stage 5 completed, Stage 6 next 2026-08-23 17:58:41 +00:00
NSCT Agent
e8b6515f67 feat(stage5): implement claim extraction — atomic verifiable claims from sources 
- Claim model with provenance, evidence_span, attribution, claim_type
- Stage5Extractor: LLM-based atomic claim extraction from source content
  - Never summarizes — always extracts atomic, verifiable claims
  - Claims require evidence span (exact quote from source)
  - Attribution per claim (who says what)
  - Claim types: fact, opinion, prediction, recommendation, claim
  - Confidence score 0.0–1.0 per claim
  - Bounded concurrency, SSRF-safe, max content truncation
- REST API: GET/POST /research/{run_id}/claims
- 36 tests: parsing, edge cases, integration, validation
 2026-08-23 17:56:01 +00:00
NSCT Agent
b8181deb05 Stage 4: Research Planner — LLM-basierte Recherchestrategie-Generierung 
- src/nsct/agents/planner.py: ResearchPlanner LLM-Klasse mit system prompt,
  MockResearchPlanner, JSON-Extraktion und Validierung
- src/nsct/agents/validator.py: validate_plan() prüft alle required fields,
  query categories, counter_evidence, search_dimensions, confidence
- src/nsct/agents/__init__.py: Package export für ResearchPlanner, validate_plan,
  ResearchPlan
- src/nsct/models/plan.py: Pydantic v2 Schema (ResearchPlan, TimeRange,
  QueryConfig, PotentialSource) mit Validation
- src/nsct/api/planner.py: POST /research/planner Endpoint mit Debug-Support
- src/nsct/api/main.py: Mount des planner routers
- src/nsct/crawler/pdf.py: exportiere extract_pdf_content als Alias
- src/nsct/api/crawler.py: Pydantic BaseModel für Request-Models
- tests/test_planner.py: 25 Tests für Planner, Validator, Schema, API

- Search-Bias-Reduktion: 6+ Query-Typen, counter_evidence, beide Seiten
- Keine TODOs, keine unvollständigen Funktionen
- Alle Dateien syntaktisch korrekt und getestet
 2026-08-23 13:28:23 +00:00
NSCT Agent
a8595cc950 Stage 3: Crawler und Content Extraction 
- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
  (50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
  content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
  Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
  localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
  /crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
  Error Handling, Content Hash Determinismus
 2026-08-23 12:53:19 +00:00
NSCT Agent
a1ef260520 STAGE 2: Search Provider Abstraction für NSCT 
- SearchProvider-Interface mit abstract.base, NormalizedResult-Modell
- DuckDuckGoProvider: HTTP-basierte Suche ohne API-Keys, Fallback-fähig
- MultiProviderSearch: parallele Suche, URL-Dedup, Provider-Config, Fallback
- POST /search-Endpoint mit normalisierten Ergebnissen, Debug-Mode
- 25 unit tests: NormalizedResult, MultiProviderSearch, DuckDuckGoProvider
- rank ist KEIN truth_score - Dokumentation und Validierung durchgängig
 2026-08-23 12:16:29 +00:00
NSCT Agent
9280d69ebf Stage 1: OpenAI-compatible provider layer (llm, vision, audio, metrics, debug) 2026-08-23 11:54:50 +00:00
NSCT Agent
e9410be941 Stage 0: Repository und Architekturgrundlage 
- Pyproject.toml mit FastAPI, Pydantic v2, SQLAlchemy, httpx, asyncio,
  BeautifulSoup4, selectolax, trafilatura, uvicorn, pytest-asyncio
- Multi-stage Dockerfile (Python 3.12-slim, Non-Root-User nsct)
- docker-compose.yml (nsct-api + postgres + optional searxng)
- .env.example mit allen Config-Parametern
- Config-System: AppSettings mit LLMConfig, VisionConfig, AudioConfig,
  DatabaseConfig — komplett aus Environment, keine Hardcodes
- Strukturiertes Logging mit research_id/llm_request_id Tracking
- Pydantic v2 Schemas: SearchQuery, Source, Claim, EvidenceRelation,
  CitationEdge, ResearchReport
- SQLAlchemy 2.0 Declarative Models + async Engine Factory
- SSRF-Schutz: URL-Validation, IP-Blocklist (RFC1918, Cloud Metadata,
  file://, ftp://)
- Provider-Interfaces: LLMProvider, VisionProvider, AudioProvider,
  SearchProvider, ContentFetcher als ABCs
- Health-Endpoints: /health, /ready (LLM-Connect-Test), /providers
- FastAPI App mit CORS, lifespan (LLM Pre-Flight)
- CLI-Stub mit Entry-Points: nsct, nsct-core, nsct-api
- 6 Test-Cases: /health, /ready, /providers + No-Secrets-Test
- Vollständige Dokumentation: README, ARCHITECTURE, SECURITY,
  METHODOLOGY, API, DEPLOYMENT
- .gitignore (Python, Docker, IDE, .env)
 2026-08-23 11:33:45 +00:00