c8cb4cbbe4
Replace homepage-only crawling (the reason the index had ~9k docs and useless results) with a real crawler: - SQLite frontier (node:sqlite) — durable queue, seen-set, per-domain caps - follows internal links to configurable depth, seeds from sitemap.xml - robots.txt compliance + per-host politeness delay - per-URL doc IDs (old id=domain overwrote every page of a site) - link graph: cross-domain in-degree recorded as an authority signal - authority score field + 'score:desc' ranking rule (relevance-first, authority tiebreak) - body trimmed for the disk-bound VM Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
14 lines
289 B
JSON
14 lines
289 B
JSON
{
|
|
"name": "search-engine-crawler",
|
|
"version": "1.0.0",
|
|
"type": "module",
|
|
"scripts": {
|
|
"start": "node --experimental-sqlite src/index.js",
|
|
"tranco": "node src/tranco.js"
|
|
},
|
|
"dependencies": {
|
|
"cheerio": "^1.0.0",
|
|
"p-limit": "^6.2.0",
|
|
"undici": "^7.0.0"
|
|
}
|
|
}
|