NSCT Agent
|
a8595cc950
|
Stage 3: Crawler und Content Extraction
- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
(50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
/crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
Error Handling, Content Hash Determinismus
|
2026-08-23 12:53:19 +00:00 |
|
NSCT Agent
|
a1ef260520
|
STAGE 2: Search Provider Abstraction für NSCT
- SearchProvider-Interface mit abstract.base, NormalizedResult-Modell
- DuckDuckGoProvider: HTTP-basierte Suche ohne API-Keys, Fallback-fähig
- MultiProviderSearch: parallele Suche, URL-Dedup, Provider-Config, Fallback
- POST /search-Endpoint mit normalisierten Ergebnissen, Debug-Mode
- 25 unit tests: NormalizedResult, MultiProviderSearch, DuckDuckGoProvider
- rank ist KEIN truth_score - Dokumentation und Validierung durchgängig
|
2026-08-23 12:16:29 +00:00 |
|
NSCT Agent
|
e9410be941
|
Stage 0: Repository und Architekturgrundlage
- Pyproject.toml mit FastAPI, Pydantic v2, SQLAlchemy, httpx, asyncio,
BeautifulSoup4, selectolax, trafilatura, uvicorn, pytest-asyncio
- Multi-stage Dockerfile (Python 3.12-slim, Non-Root-User nsct)
- docker-compose.yml (nsct-api + postgres + optional searxng)
- .env.example mit allen Config-Parametern
- Config-System: AppSettings mit LLMConfig, VisionConfig, AudioConfig,
DatabaseConfig — komplett aus Environment, keine Hardcodes
- Strukturiertes Logging mit research_id/llm_request_id Tracking
- Pydantic v2 Schemas: SearchQuery, Source, Claim, EvidenceRelation,
CitationEdge, ResearchReport
- SQLAlchemy 2.0 Declarative Models + async Engine Factory
- SSRF-Schutz: URL-Validation, IP-Blocklist (RFC1918, Cloud Metadata,
file://, ftp://)
- Provider-Interfaces: LLMProvider, VisionProvider, AudioProvider,
SearchProvider, ContentFetcher als ABCs
- Health-Endpoints: /health, /ready (LLM-Connect-Test), /providers
- FastAPI App mit CORS, lifespan (LLM Pre-Flight)
- CLI-Stub mit Entry-Points: nsct, nsct-core, nsct-api
- 6 Test-Cases: /health, /ready, /providers + No-Secrets-Test
- Vollständige Dokumentation: README, ARCHITECTURE, SECURITY,
METHODOLOGY, API, DEPLOYMENT
- .gitignore (Python, Docker, IDE, .env)
|
2026-08-23 11:33:45 +00:00 |
|