Replace homepage-only crawling (the reason the index had ~9k docs and
useless results) with a real crawler:
- SQLite frontier (node:sqlite) — durable queue, seen-set, per-domain caps
- follows internal links to configurable depth, seeds from sitemap.xml
- robots.txt compliance + per-host politeness delay
- per-URL doc IDs (old id=domain overwrote every page of a site)
- link graph: cross-domain in-degree recorded as an authority signal
- authority score field + 'score:desc' ranking rule (relevance-first, authority tiebreak)
- body trimmed for the disk-bound VM
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
