Add SHARD_ID/SHARD_COUNT env: each worker seeds only domains where
hash(domain) % SHARD_COUNT === SHARD_ID (FNV-1a). Defaults to single-node
(all domains). Crawler only follows same-domain links, so shards never
overlap and lose no coverage vs a single crawler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crawler now records the deduped cross-domain link graph (edges table).
recompute.js runs real iterative PageRank over that graph, blends it with
accumulated in-degree + the Majestic seed prior, and patches every doc's
`score` (Meili's authority tie-breaker) via partial updates — preserving
titles/bodies/embeddings and triggering no re-embedding. systemd timer runs
it every 3h so ranking sharpens as the 1.5M-page crawl densifies the graph.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-host politeness serialized entire same-host batches (contiguous internal
links at depth), collapsing throughput to ~1 request in flight. Randomize the
frontier claim so each batch spans many hosts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The crawl stalled when two crawler processes briefly overlapped on the same
WAL SQLite frontier (node:sqlite is synchronous, so a blocked write lock froze
the event loop). Fixes:
- run the crawler as a single-instance systemd service (search-crawler.service)
with CPUQuota=150% + MemoryMax=1200M so it can't starve a shared VM
- add timeouts to all Meilisearch calls + resilient flush (an untimed hang was
the amplifier that wedged every worker)
- crawl-diskguard.sh + cron: auto-stop the crawl and ntfy-alert if disk < 800MB
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace homepage-only crawling (the reason the index had ~9k docs and
useless results) with a real crawler:
- SQLite frontier (node:sqlite) — durable queue, seen-set, per-domain caps
- follows internal links to configurable depth, seeds from sitemap.xml
- robots.txt compliance + per-host politeness delay
- per-URL doc IDs (old id=domain overwrote every page of a site)
- link graph: cross-domain in-degree recorded as an authority signal
- authority score field + 'score:desc' ranking rule (relevance-first, authority tiebreak)
- body trimmed for the disk-bound VM
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crawler (Majestic Million → undici → cheerio → Meilisearch), Hono search
API with P2P peer federation, Cloudflare Worker frontend, VM provisioning
scripts. Adds README, .gitignore, and moves VM credentials out of deploy.sh
into a git-ignored deploy.env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
