c8cb4cbbe4
Replace homepage-only crawling (the reason the index had ~9k docs and useless results) with a real crawler: - SQLite frontier (node:sqlite) — durable queue, seen-set, per-domain caps - follows internal links to configurable depth, seeds from sitemap.xml - robots.txt compliance + per-host politeness delay - per-URL doc IDs (old id=domain overwrote every page of a site) - link graph: cross-domain in-degree recorded as an authority signal - authority score field + 'score:desc' ranking rule (relevance-first, authority tiebreak) - body trimmed for the disk-bound VM Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
21 lines
263 B
Text
21 lines
263 B
Text
# Dependencies
|
|
node_modules/
|
|
|
|
# Crawler data
|
|
crawler/domains.csv
|
|
crawler/*.csv
|
|
crawler/frontier.db*
|
|
|
|
# Wrangler / Cloudflare
|
|
.wrangler/
|
|
dist/
|
|
|
|
# Local env / secrets (never commit VM passwords, keys)
|
|
.env
|
|
.env.local
|
|
deploy.env
|
|
*.local
|
|
|
|
# Logs & OS
|
|
*.log
|
|
.DS_Store
|