maverick/search-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
search-engine/crawler
History
Exact
King Omar 7fa8114a7f Crawler: shard-aware seeding for distributed crawling 
Add SHARD_ID/SHARD_COUNT env: each worker seeds only domains where
hash(domain) % SHARD_COUNT === SHARD_ID (FNV-1a). Defaults to single-node
(all domains). Crawler only follows same-domain links, so shards never
overlap and lose no coverage vs a single crawler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 23:17:35 +10:00
..
src Crawler: shard-aware seeding for distributed crawling 2026-07-22 23:17:35 +10:00
package.json Overhaul crawler: depth crawl, link graph, robots, sitemaps, authority 2026-07-22 02:26:22 +10:00