The crawl stalled when two crawler processes briefly overlapped on the same
WAL SQLite frontier (node:sqlite is synchronous, so a blocked write lock froze
the event loop). Fixes:
- run the crawler as a single-instance systemd service (search-crawler.service)
with CPUQuota=150% + MemoryMax=1200M so it can't starve a shared VM
- add timeouts to all Meilisearch calls + resilient flush (an untimed hang was
the amplifier that wedged every worker)
- crawl-diskguard.sh + cron: auto-stop the crawl and ntfy-alert if disk < 800MB
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A deployed Worker's fetch() cannot reach the API on :3000, so publish it at
https://search-api.theradicalparty.com through a Cloudflare Tunnel. Point the
frontend API_URL at HTTPS/443, add cloudflared install to install-vm.sh
(driven by CLOUDFLARED_TOKEN), thread the token through deploy.sh, and add
docs/DEPLOY.md with the full tunnel + DNS setup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crawler (Majestic Million → undici → cheerio → Meilisearch), Hono search
API with P2P peer federation, Cloudflare Worker frontend, VM provisioning
scripts. Adds README, .gitignore, and moves VM credentials out of deploy.sh
into a git-ignored deploy.env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
