opencode/NSCT---Neutral-Search-Crawler-Tool
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
56 Commits 1 Branch 0 Tags
a5324d39711e683817c7dd0afe7ce96ec01409e5
Commit Graph

56 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
faligam
a5324d3971 Fix research pipeline runtime failures 2026-09-07 11:16:36 +02:00
faligam
cf10cd9636 Update handoff for frontend deletion fix 2026-09-06 18:36:53 +02:00
faligam
e5b7d90436 Fix research pipeline provider failures 2026-09-06 18:32:42 +02:00
faligam
9f6029b60a Document browser flow backend blockers 2026-09-06 17:59:16 +02:00
faligam
4f8cbe9040 Document research detail browser flow fixes 2026-09-06 17:52:29 +02:00
faligam
73b69fecb5 Document research route parameter fix 2026-09-06 17:48:24 +02:00
faligam
9c83f305b0 Document dynamic research route fix 2026-09-06 17:45:19 +02:00
faligam
7c41f9bdde Document browser session and API contract fixes 2026-09-06 17:39:44 +02:00
faligam
b0aaddd236 Document browser flow error banner fix 2026-09-06 17:29:53 +02:00
faligam
fd1250ce2e Update handoff for deployed API-key testing 
Record the backend and frontend authentication commits, the rebuilt local stacks, and the next browser-to-backend end-to-end test procedure.
 2026-09-06 17:25:09 +02:00
faligam
1aacf4aa20 Implement persistent API-key authentication 
Protect the research lifecycle with X-API-Key validation backed by persistent user and key records. Store only salted scrypt hashes, support expiry and revocation, and expose a local admin CLI for create/list/revoke workflows.

Initialize only the authentication schema at startup, prevent SQL echo from exposing sensitive bound values, and keep health probes public. Add coverage for valid, missing, invalid, expired, and revoked keys.

Document deployment and key administration, update the local CLI to send NSCT_API_KEY, and record the reset handoff state.
 2026-09-06 17:23:05 +02:00
faligam
64eb4e8465 Update reset handoff for API authentication work 2026-09-06 16:59:34 +02:00
faligam
9830acf446 Document remote frontend integration 2026-09-06 16:55:48 +02:00
faligam
fddb4da1c7 Update ADMIN-Backend.md 2026-09-06 14:24:51 +00:00
faligam
ac905e7198 Fix Docker installation startup 2026-09-06 13:01:01 +02:00
NSCT Agent
8a918c9d86 Fix: Replace GitHub URL with git.frerkc.de in Admin Handbook 2026-09-06 09:52:26 +00:00
NSCT Agent
8276975e93 Admin Handbook: Backend (NSCT API) 2026-09-06 08:37:06 +00:00
NSCT Agent
9ae1a7ba21 stage22: Abschluss & Production Readiness — E2E-Tests, CHANGELOG, Dokumentation 2026-09-05 15:03:48 +00:00
NSCT Agent
8ea6269f9a feat(stage21): implement reproducibility — research_run_hash, full provenance tracking 
- compute_research_run_hash(): deterministic SHA256 from query+budget+created_at
- ProvenanceEntry / ProvenanceLog: step-by-step trace with inputs, outputs, metadata, errors
- ResearchRun model: research_run_hash field (64-char hex)
- ResearchRunProvenance SQLAlchemy table for DB persistence
- Orchestrator: auto-log provenance before/after each pipeline step
- 35 tests: hash determinism, entry/log methods, storage model, integration
 2026-09-05 14:27:45 +00:00
NSCT Agent
f9b761ced7 fix: priority_queue async context methods (async_fix for Python 3.14+). Add test_priority_limiter.py tests for async context manager 2026-09-05 13:36:49 +00:00
NSCT Agent
6e2e7386ad stage20: context budgeting - per-stage token limits (planner 12k, claim 16k, contradiction 24k, synthesis 48k) 
Implemented:
- context_budget.py: ContextBudgetConfig (Pydantic, frozen) mit 4 Stage-Limits,
  validation (ge/le), get_limit(), total_max_tokens, stage_keys
- context_budget.py: ContextBudgetTracker mit track_tokens(), get_usage(),
  is_exhausted(), reset_stage(), reset_all(), total_usage, elapsed_seconds
- context_budget.py: ContextBudgetExhaustedError mit stage_name, used_tokens, limit_tokens
- orchestrator.py: _track_context_tokens() Methode, context_budget_config/tracker init
- orchestrator.py: context_budget_tracker property export
- __init__.py: exports ContextBudgetConfig, ContextBudgetExhaustedError, ContextBudgetTracker
- priority_queue.py: __aenter__/__aexit__ auf async geandert (Testfix)
- test_context_budget.py: 28 Tests (DefaultConfig, TrackAccumulation, ExhaustedError,
  GetUsage, Reset, IsExhausted, InvalidStageNames, InvalidTokens)
- test_context_budget_integration.py: 6 Tests (Orchestrator-Integration)
- HANDOFF.md: Stage 20 abgeschlossen dokumentiert

Tests: 34 passed (28 unit + 6 integration) + 14 performance = 48 total
 2026-08-29 08:57:18 +00:00
NSCT Agent
58488de7b3 docs: update HANDOFF.md — Stage 19 completed, 20 next 2026-08-28 17:32:32 +00:00
NSCT Agent
4335a40d68 stage19: performance optimierung — LLM concurrency semaphore, priority (HIGH/NORMAL/LOW), limiter 2026-08-28 17:28:31 +00:00
NSCT Agent
ce9b031ea0 docs: update HANDOFF.md — Stage 18 completed, 19 next 2026-08-28 16:23:43 +00:00
NSCT Agent
60679083fa stage18: docker hardening — non-root user, read-only FS, no-new-privileges, health-check fix, .dockerignore 2026-08-28 16:22:23 +00:00
NSCT Agent
b23ff5a565 docs: update HANDOFF.md — Stages 14-16 completed, 17 next 2026-08-27 20:35:40 +00:00
NSCT Agent
413050bdd9 docs: update HANDOFF.md — Stage 16 completed, Stage 17 next 2026-08-27 19:52:04 +00:00
NSCT Agent
08ec0e290c feat(stage16): implement observability — structured logging, metrics, tracing 
- JSONFormatter, ContextVars, set_research_run_id() with short logger names
- self-built Counter/Histogram/Gauge system (no external deps)
- Prometheus text export at /metrics
- Request logging middleware with X-Request-ID
- Metrics instrumentation: search_queries, sources_fetched, claims, contradictions
- Histograms: research_duration, llm_request_duration
- Gauge: active_research_runs
- 13 + 21 = 34 tests
 2026-08-27 19:44:46 +00:00
NSCT Agent
b2340fca63 docs: update HANDOFF.md — Stage 15 completed, Stage 16 next 2026-08-27 17:11:29 +00:00
NSCT Agent
1cc17cf9ac feat(stage15): implement CLI with search, status, report, sources, claims, list, delete 
- argparse subcommands: help, search, status, report, sources, claims, list, delete
- httpx client with NSCT_API_URL env var (default http://localhost:8080)
- ANSI colored output: green/completed, red/failed, yellow/running
- search command with --depth (quick/normal/deep) and --language (de/en)
- Polling loop for search with Ctrl+C interrupt
- ASCII table formatting for sources/claims/list
- Connection error handling with clear messages
- 28 tests for all commands and helper functions
 2026-08-27 17:09:36 +00:00
NSCT Agent
c059fb0ffd docs: update HANDOFF.md — Stage 14 completed, Stage 15 next 2026-08-27 16:14:51 +00:00
NSCT Agent
4d5f6747f7 feat(stage14): implement REST API for research lifecycle 
- POST /v1/research — start research (non-blocking, background pipeline)
- GET  /v1/research/{id} — research metadata
- GET  /v1/research/{id}/status — detailed state machine status
- GET  /v1/research/{id}/sources — sources list
- GET  /v1/research/{id}/claims — claims list
- GET  /v1/research/{id}/evidence — evidence scores
- GET  /v1/research/{id}/report — research report
- DELETE /v1/research/{id} — delete research (non-completed)
- GET  /v1/research — paginated list of all research runs

Depth budgets (quick/normal/deep) control only resource limits.
In-memory store for now, to be replaced with PostgreSQL later.
Router mounted in main.py as tag "research-api".
 2026-08-27 16:07:47 +00:00
Frerk Campen
a55e26f3b1 docs: update HANDOFF.md — Stage 13 completed, Stage 14 next 2026-08-26 11:22:02 +00:00
Frerk Campen
0b7a624bc8 feat(stage13): implement Iterative Research / Gap Analysis\n\nImplement Gap Analysis Engine (Stage 13):\n- GapAnalysisEngine: detect single-source claims, contradictions,\n missing primary sources, weak evidence\n- IterationReport: structured gap findings with severity & target\n- GapSearchQuery: derived search queries per gap finding\n- Integration into ResearchOrchestrator: runs gap analysis after\n extracting, then executes gap searches iteratively\n- 17 tests covering all analysis categories and edge cases 2026-08-26 11:19:35 +00:00
NSCT Agent
2ef7b67002 feat(stage12): implement Research Orchestrator 2026-08-26 10:29:57 +00:00
NSCT Agent
825aedb057 feat(stage11): implement audio integration 2026-08-25 18:37:31 +00:00
NSCT Agent
4b8ae6a41a fix(stage11): resolve subagent merge conflicts — audio models, vision fix, test fixes 2026-08-25 17:29:17 +00:00
NSCT Agent
d76d48a0ee docs: update HANDOFF.md — Stages 10+11 completed, Stage 12 next 2026-08-25 15:42:00 +00:00
NSCT Agent
10a083aa02 feat(stage11): audio integration — STT for interviews, podcasts, press conferences with timestamped claims 2026-08-25 15:31:03 +00:00
NSCT Agent
3ab875d2bc feat(stage10): implement vision integration 2026-08-25 14:42:29 +00:00
NSCT Agent
14f016457a tests(stage9): fix test_stage9_synthesis.py 2026-08-25 14:08:15 +00:00
NSCT Agent
8582a8e60f docs: update HANDOFF.md — Stage 9 completed, Stage 10 next 2026-08-24 12:38:47 +00:00
NSCT Agent
d87e2b4d14 feat(stage9): neutral synthesis engine — LLM-generated report from evidence package 2026-08-24 11:52:10 +00:00
NSCT Agent
1b54b172ca docs: update HANDOFF.md — Stage 8 completed, Stage 9 next 2026-08-24 07:52:43 +00:00
NSCT Agent
719e218d9a feat(stage8): evidence scoring — transparent multidimensional scores (independence, proximity, support, contradiction, directness, date) 2026-08-24 07:47:00 +00:00
NSCT Agent
cc9d4f2ba5 docs: update HANDOFF.md — Stage 7 completed, Stage 8 next 2026-08-23 21:00:40 +00:00
NSCT Agent
d1e6bb6cf4 feat(stage7): claim clustering & contradiction candidates — semantic grouping, numeric normalization, pairwise analysis 
- ClaimClusterModel: LLM-basierte semantische Gruppierung von Claims
- ClaimRelationModel: SUPPORTS, CONTRADICTS, DUPLICATE, UNCERTAIN pairwise relations
- ClaimNLUModel: numerische Normalisierung (%, Währungen, deutsche/englische Wörter)
- stage7_normalize_numerics.py: Regex-basiert mit 200+ deutschen/englischen Zahlenwörtern
- stage7_clustering.py: LLM-Clustering + pairwise claim-relation analysis
- API: POST cluster-claims, GET clusters, GET claim-relations
- 79 tests: numerische Normalisierung, LLM-Parsing, Clustering, Relationen, Edge-Cases
- Dedup: claims mit gleichen numerischen Werten werden zusammengefasst
 2026-08-23 20:57:17 +00:00
NSCT Agent
d2294c58d0 docs: update HANDOFF.md — Stage 6 completed, Stage 7 next 2026-08-23 18:50:25 +00:00
NSCT Agent
a60cf21a2c feat(stage6): source independence & citation graph — detect syndication, shared origins, text similarity 
- SourceIndependenceModel: per-source independence_score (0.0-1.0), syndication_group_id, primary_source_id, content_hash, shared_urls
- CitationGraphEdgeModel: directed edges (SYNDICATED, QUOTES, LINKS_TO, REPOST, SIMILAR_CONTENT) with confidence + evidence
- Content-Hash (SHA-256): instant syndication detection for identical content
- difflib Vorfilterung: >60% → LLM, >80% → high confidence, 100% → immediate syndication
- LLM-Pairwise-Analysis: two-text-comparison for suspicious pairs only (bounded concurrency)
- independence_score: 1.0 base, -0.4 for syndicated, -0.1 per high-similarity pair
- Pydantic schemas: SourceIndependenceScore, CitationGraphEdge, SyntacticSimilarityResult, LlmSyndicationAnalysis, SourceIndependenceAnalysisResult
- LLM response parser: handles JSON, markdown code blocks, partial/invalid JSON
- 43 tests: content hash, similarity thresholds, LLM parsing, analyzer integration, edge cases, prompt templates
 2026-08-23 18:47:24 +00:00
NSCT Agent
27e494d161 docs: update HANDOFF.md — Stage 5 completed, Stage 6 next 2026-08-23 17:58:41 +00:00