Stage 3: Crawler und Content Extraction

- AsyncFetcher: Sicheres HTTP-Fetching mit SSRF-Schutz, Connection Pooling
  (50/10), Timeout, Redirect Limit, Rate Limiting, User-Agent
- Content-Extraction: trafilatura für HTML→Text, BeautifulSoup4 Fallback
- PDF-Extraction: pdfminer.six mit Error-Handling
- NormalizedDocument: Schema (url, title, text, metadata, links,
  content_hash, extraction_tool, extracted_at, word_count)
- Crawler-Manager: SSRF-Check → robots.txt → HTTP-Fetch → Extraction →
  Normalization (Batch-fähig, Error-Isolation pro Fetch)
- Security-Policy: SSRF-Schutz (RFC1918, Cloud Metadata, file://, ftp://,
  localhost), URL-Validation (nur http/https)
- Crawler-Endpoints: POST /crawler/fetch, /crawler/fetch/batch,
  /crawler/validate-url
- Test-Cases: SSRF-Schutz, Content Extraction, NormalizedDocument,
  Error Handling, Content Hash Determinismus

src/nsct/api/main.py

@@ -79,6 +79,10 @@ def create_app() -> FastAPI:
     from nsct.api.search import router as search_router
     app.include_router(search_router, tags=["search"])
 
+    # Mount crawler router
+    from nsct.api.crawler import router as crawler_router
+    app.include_router(crawler_router, tags=["crawler"])
+
     return app