maverick/search-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix crawl stall: host-diverse claim (ORDER BY RANDOM)

Browse source 
Per-host politeness serialized entire same-host batches (contiguous internal
links at depth), collapsing throughput to ~1 request in flight. Randomize the
frontier claim so each batch spans many hosts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
King Omar 2026-07-22 13:22:36 +10:00
parent ab6233ddbd
commit 538f7a55bc
1 changed files with 3 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
crawler/src/frontier.js
View file

@ -22,7 +22,9 @@ db.exec(`
const q = {
add: db.prepare('INSERT OR IGNORE INTO frontier(url,domain,depth,rank) VALUES(?,?,?,?)'),
next: db.prepare('SELECT url,domain,depth,rank FROM frontier WHERE status=0 LIMIT ?'),
// Randomize the claim so a batch spans many hosts — otherwise contiguous
// same-host internal links get serialized by per-host politeness (stall).
next: db.prepare('SELECT url,domain,depth,rank FROM frontier WHERE status=0 ORDER BY RANDOM() LIMIT ?'),
claim: db.prepare('UPDATE frontier SET status=3 WHERE url=?'),
done: db.prepare('UPDATE frontier SET status=1 WHERE url=?'),
fail: db.prepare('UPDATE frontier SET status=2 WHERE url=?'),