King Omar 538f7a55bc Fix crawl stall: host-diverse claim (ORDER BY RANDOM) ... Per-host politeness serialized entire same-host batches (contiguous internal links at depth), collapsing throughput to ~1 request in flight. Randomize the frontier claim so each batch spans many hosts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>		2026-07-22 13:22:36 +10:00
..
src	Fix crawl stall: host-diverse claim (ORDER BY RANDOM)	2026-07-22 13:22:36 +10:00
package.json	Overhaul crawler: depth crawl, link graph, robots, sitemaps, authority	2026-07-22 02:26:22 +10:00