Files
NSCT---Neutral-Search-Crawl…/tests
faligam 56d6415e68 fix(crawler): prioritize usable source retrieval over equal byte splits
Fetch candidate pages sequentially with the full remaining download budget instead of dividing a quick run's 500 KB equally across every URL. This avoids aborting otherwise readable pages at roughly 33 KB before claim extraction can inspect them.

Account for bytes consumed by failed downloads, preserve fetch errors in source responses, and fail runs with no successfully extracted document using an actionable error. Keep attempted sources on failed runs for diagnostics and cover the allocation and no-usable-source paths with regression tests.
2026-09-07 14:04:48 +02:00
..