Initial commit: RADICAL_SEARCH self-hosted web search engine

Crawler (Majestic Million → undici → cheerio → Meilisearch), Hono search API with P2P peer federation, Cloudflare Worker frontend, VM provisioning scripts. Adds README, .gitignore, and moves VM credentials out of deploy.sh into a git-ignored deploy.env. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:52:31 +10:00 · 2026-07-22 01:52:31 +10:00 · b3aea18d9c
commit b3aea18d9c
17 changed files with 2471 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,20 @@
 # Dependencies
 node_modules/
 # Crawler data
 crawler/domains.csv
 crawler/*.csv
 # Wrangler / Cloudflare
 .wrangler/
 dist/
 # Local env / secrets (never commit VM passwords, keys)
 .env
 .env.local
 deploy.env
 *.local
 # Logs & OS
 *.log
 .DS_Store
--- a/README.md
+++ b/README.md
@ -0,0 +1,94 @@
 # 🔍 RADICAL_SEARCH
 > **The open web — unfiltered.** A self-hosted, from-scratch web search engine.
 RADICAL_SEARCH crawls the most-linked domains on the web, indexes them in
 [Meilisearch](https://www.meilisearch.com/), and serves results through a
 Cloudflare Worker frontend at **[search.theradicalparty.com](https://search.theradicalparty.com)**.
 The search API also supports optional **P2P federation** — nodes can register as
 peers and fan queries out across a cluster.
 ## Architecture
 ```
                    ┌─────────────────────────────┐
   browser  ───▶    │  frontend/  (Cloudflare Worker)  │  search.theradicalparty.com
                    │  SSR UI, dark/light, proxies /search │
                    └───────────────┬─────────────┘
                                    │  fetch(API_URL)  (HTTPS :443)
                                    ▼
                    ┌─────────────────────────────┐
                    │  api/  (Hono, Node, on VM)   │  search-api.theradicalparty.com
                    │  /search /peers /stats /health │
                    └───────────────┬─────────────┘
                                    │  localhost:7700
                                    ▼
                    ┌─────────────────────────────┐
                    │  Meilisearch  (index: pages) │
                    └───────────────▲─────────────┘
                                    │  batched docs
                    ┌───────────────┴─────────────┐
                    │  crawler/  (Node, on VM)     │  Majestic Million → fetch → parse → index
                    └─────────────────────────────┘
 ```
 ## Components
 | Dir | What it is | Runtime |
 |-----|-----------|---------|
 | `crawler/` | Downloads the [Majestic Million](https://majestic.com/reports/majestic-million) domain list, crawls each domain (undici, 8s timeout, HTML-only, 500 KB cap), parses with cheerio (title/description/body/outlinks), and batches docs into Meilisearch. | Node on the VM |
 | `api/` | Hono search API. `/search` proxies to Meilisearch with highlighting and optional P2P peer fan-out. Also `/peers`, `/peers/register`, `/stats`, `/health`. | Node on the VM (`:PORT`) |
 | `frontend/` | Cloudflare Worker. Server-rendered UI (Roboto Mono, dark/light theme), proxies `/search` to `API_URL`. | Cloudflare Workers |
 | `scripts/` | `install-vm.sh` provisions Meilisearch + api + crawler as systemd services; `deploy.sh` syncs & runs it over SSH. | local → VM |
 ## Local development
 ```bash
 # 1. Meilisearch (Docker)
 docker run -it --rm -p 7700:7700 getmeili/meilisearch:latest --master-key masterKey
 # 2. API
 cd api && npm install && MEILI_KEY=masterKey npm run dev      # :3000
 # 3. A small crawl
 cd crawler && npm install && LIMIT=500 CONCURRENCY=8 npm start
 # 4. Frontend
 cd frontend && npm install && npm run dev                     # wrangler dev
 ```
 ## Deploy
 **Frontend (Cloudflare Worker):**
 ```bash
 cd frontend && npm run deploy
 ```
 **API + crawler (VM):** copy `scripts/deploy.env.example` → `scripts/deploy.env`,
 fill in the VM host/user/password (this file is git-ignored), then:
 ```bash
 cd scripts && ./deploy.sh
 ```
 ### API exposure
 The frontend Worker reaches the API over **HTTPS on port 443** — a deployed
 Cloudflare Worker's `fetch()` cannot reliably use non-standard ports like 3000.
 The API is fronted at `search-api.theradicalparty.com` (see
 [`docs/DEPLOY.md`](docs/DEPLOY.md) for the port/DNS setup).
 ## Configuration
 | Var | Component | Default | Notes |
 |-----|-----------|---------|-------|
 | `MEILI_URL` | api, crawler | `http://localhost:7700` | Meilisearch endpoint |
 | `MEILI_KEY` | api, crawler | `masterKey` | Meilisearch master key (VM-local only) |
 | `PORT` | api | `3000` | API listen port |
 | `NODE_ID` | api | `main` | P2P node identity |
 | `CONCURRENCY` | crawler | `8` | Parallel fetches |
 | `LIMIT` | crawler | `100000` | Max domains to crawl |
 | `API_URL` | frontend | — | Set in `frontend/wrangler.jsonc` `vars` |
 ## License
 Private / unlicensed — © the maintainer.
--- a/api/package.json
+++ b/api/package.json
@ -0,0 +1,13 @@
 {
  "name": "search-engine-api",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node src/index.js",
    "dev": "node --watch src/index.js"
  },
  "dependencies": {
    "hono": "^4.7.0",
    "@hono/node-server": "^1.14.0"
  }
 }
--- a/api/src/index.js
+++ b/api/src/index.js
@ -0,0 +1,115 @@
 import { Hono } from 'hono'
 import { serve } from '@hono/node-server'
 const app = new Hono()
 const PORT = parseInt(process.env.PORT || '3000')
 const MEILI_URL = process.env.MEILI_URL || 'http://localhost:7700'
 const MEILI_KEY = process.env.MEILI_KEY || 'masterKey'
 const NODE_ID = process.env.NODE_ID || 'main'
 // In-memory peer registry  { id, url, lastSeen }
 const peers = new Map()
 async function meiliSearch(q, offset = 0, limit = 10) {
  const res = await fetch(`${MEILI_URL}/indexes/pages/search`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${MEILI_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      q,
      limit,
      offset,
      attributesToHighlight: ['title', 'description'],
      highlightPreTag: '<mark>',
      highlightPostTag: '</mark>',
      attributesToRetrieve: ['url', 'domain', 'title', 'description', 'crawledAt'],
    }),
  })
  return res.json()
 }
 async function queryPeer(peer, q) {
  try {
    const res = await fetch(`${peer.url}/search?q=${encodeURIComponent(q)}&local=1`, {
      signal: AbortSignal.timeout(3000),
      headers: { 'X-Peer-Id': NODE_ID },
    })
    const data = await res.json()
    return data.hits ?? []
  } catch {
    return []
  }
 }
 // Search endpoint — fans out to peers when not local
 app.get('/search', async (c) => {
  const q = c.req.query('q')?.trim()
  const page = Math.max(0, parseInt(c.req.query('page') ?? '0'))
  const local = c.req.query('local') === '1'
  if (!q) return c.json({ error: 'missing q' }, 400)
  const offset = page * 10
  const local$ = meiliSearch(q, offset)
  const peer$ = local ? [] : [...peers.values()].map(p => queryPeer(p, q))
  const [localResult, ...peerResults] = await Promise.all([local$, ...peer$])
  // Deduplicate and merge peer hits
  const seen = new Set(localResult.hits?.map(h => h.url) ?? [])
  const peerHits = peerResults
    .flat()
    .filter(h => !seen.has(h.url))
    .slice(0, 5)
  const hits = [...(localResult.hits ?? []), ...peerHits]
  return c.json({
    hits,
    total: (localResult.estimatedTotalHits ?? 0) + peerHits.length,
    page,
    query: q,
    nodeId: NODE_ID,
    peers: [...peers.keys()],
  })
 })
 // P2P: register as peer
 app.post('/peers/register', async (c) => {
  const { id, url } = await c.req.json()
  if (!id || !url) return c.json({ error: 'id and url required' }, 400)
  peers.set(id, { id, url, lastSeen: Date.now() })
  return c.json({ ok: true, peerId: NODE_ID, peers: [...peers.keys()] })
 })
 // P2P: list peers
 app.get('/peers', (c) => {
  return c.json({ nodeId: NODE_ID, peers: [...peers.values()] })
 })
 // Health
 app.get('/health', async (c) => {
  try {
    const res = await fetch(`${MEILI_URL}/health`, {
      headers: { 'Authorization': `Bearer ${MEILI_KEY}` }
    })
    const meili = await res.json()
    return c.json({ ok: true, meili, nodeId: NODE_ID })
  } catch (e) {
    return c.json({ ok: false, error: e.message }, 503)
  }
 })
 // Stats
 app.get('/stats', async (c) => {
  const res = await fetch(`${MEILI_URL}/indexes/pages/stats`, {
    headers: { 'Authorization': `Bearer ${MEILI_KEY}` }
  })
  const stats = await res.json()
  return c.json({ ...stats, peers: peers.size, nodeId: NODE_ID })
 })
 serve({ fetch: app.fetch, port: PORT })
 console.log(`Search API running on :${PORT} [node=${NODE_ID}]`)
--- a/crawler/package.json
+++ b/crawler/package.json
@ -0,0 +1,14 @@
 {
  "name": "search-engine-crawler",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node src/index.js",
    "tranco": "node src/tranco.js"
  },
  "dependencies": {
    "cheerio": "^1.0.0",
    "p-limit": "^6.2.0",
    "undici": "^7.0.0"
  }
 }
--- a/crawler/src/fetcher.js
+++ b/crawler/src/fetcher.js
@ -0,0 +1,47 @@
 import { fetch } from 'undici'
 const TIMEOUT_MS = 8000
 const MAX_BODY_BYTES = 500_000
 const HEADERS = {
  'User-Agent': 'SearchBot/1.0 (compatible; +https://search.theradicalparty.com/bot)',
  'Accept': 'text/html,application/xhtml+xml',
  'Accept-Language': 'en',
 }
 export async function fetchPage(domain) {
  const urls = [`https://${domain}`, `http://${domain}`]
  for (const url of urls) {
    try {
      const controller = new AbortController()
      const timer = setTimeout(() => controller.abort(), TIMEOUT_MS)
      const res = await fetch(url, {
        headers: HEADERS,
        signal: controller.signal,
        maxRedirections: 3,
      })
      clearTimeout(timer)
      if (!res.ok) continue
      const contentType = res.headers.get('content-type') ?? ''
      if (!contentType.includes('html')) continue
      // Read up to MAX_BODY_BYTES
      const reader = res.body.getReader()
      const chunks = []
      let total = 0
      while (true) {
        const { done, value } = await reader.read()
        if (done) break
        chunks.push(value)
        total += value.length
        if (total >= MAX_BODY_BYTES) { reader.cancel(); break }
      }
      const html = Buffer.concat(chunks).toString('utf8')
      return { url, html, status: res.status }
    } catch {
      // try next URL
    }
  }
  return null
 }
--- a/crawler/src/index.js
+++ b/crawler/src/index.js
@ -0,0 +1,66 @@
 import pLimit from 'p-limit'
 import { downloadTranco, readDomains } from './tranco.js'
 import { fetchPage } from './fetcher.js'
 import { parsePage } from './parser.js'
 import { setupIndex, indexDoc, flush } from './indexer.js'
 const CONCURRENCY = parseInt(process.env.CONCURRENCY || '8')
 const LIMIT = parseInt(process.env.LIMIT || '100000')
 let crawled = 0
 let failed = 0
 let skipped = 0
 function log() {
  if (crawled % 50 === 0) {
    process.stdout.write(`\r[crawled=${crawled} failed=${failed} skipped=${skipped}]`)
  }
 }
 async function crawlDomain({ rank, domain }) {
  const result = await fetchPage(domain)
  if (!result) { failed++; log(); return }
  const doc = parsePage(result.url, result.html)
  if (!doc.title && !doc.description) { skipped++; log(); return }
  doc.rank = rank
  await indexDoc(doc)
  crawled++
  log()
 }
 // Process domains in a bounded sliding window instead of creating all promises upfront
 async function main() {
  await downloadTranco()
  await setupIndex()
  console.log(`Starting crawl: concurrency=${CONCURRENCY} limit=${LIMIT}`)
  const queue = []
  let active = 0
  async function runNext(entry) {
    active++
    try { await crawlDomain(entry) } catch {}
    active--
  }
  for (const entry of readDomains(LIMIT)) {
    while (active >= CONCURRENCY) {
      await new Promise(r => setTimeout(r, 10))
    }
    queue.push(runNext(entry))
    // Periodically drain settled promises to free memory
    if (queue.length >= 500) {
      await Promise.allSettled(queue.splice(0, 200))
    }
  }
  await Promise.allSettled(queue)
  await flush()
  console.log(`\nDone. crawled=${crawled} failed=${failed} skipped=${skipped}`)
 }
 main().catch(console.error)
--- a/crawler/src/indexer.js
+++ b/crawler/src/indexer.js
@ -0,0 +1,40 @@
 const MEILI_URL = process.env.MEILI_URL || 'http://localhost:7700'
 const MEILI_KEY = process.env.MEILI_KEY || 'masterKey'
 const INDEX = 'pages'
 const BATCH_SIZE = 100
 let buffer = []
 async function meili(method, path, body) {
  const res = await fetch(`${MEILI_URL}${path}`, {
    method,
    headers: {
      'Authorization': `Bearer ${MEILI_KEY}`,
      'Content-Type': 'application/json',
    },
    body: body ? JSON.stringify(body) : undefined,
  })
  return res.json()
 }
 export async function setupIndex() {
  await meili('POST', '/indexes', { uid: INDEX, primaryKey: 'id' })
  await meili('PATCH', `/indexes/${INDEX}/settings`, {
    searchableAttributes: ['title', 'description', 'domain', 'body'],
    displayedAttributes: ['id', 'url', 'domain', 'title', 'description', 'crawledAt'],
    rankingRules: ['words', 'typo', 'proximity', 'attribute', 'sort', 'exactness'],
    filterableAttributes: ['lang', 'domain'],
  })
  console.log('Index ready')
 }
 export async function indexDoc(doc) {
  buffer.push(doc)
  if (buffer.length >= BATCH_SIZE) await flush()
 }
 export async function flush() {
  if (buffer.length === 0) return
  const docs = buffer.splice(0)
  await meili('POST', `/indexes/${INDEX}/documents`, docs)
 }
--- a/crawler/src/parser.js
+++ b/crawler/src/parser.js
@ -0,0 +1,43 @@
 import { load } from 'cheerio'
 export function parsePage(url, html) {
  const $ = load(html)
  // Remove noise
  $('script, style, noscript, nav, footer, header, aside, [aria-hidden=true]').remove()
  const title = ($('title').text() || $('h1').first().text()).trim().slice(0, 200)
  const description = (
    $('meta[name=description]').attr('content') ||
    $('meta[property="og:description"]').attr('content') ||
    $('p').first().text()
  ).trim().slice(0, 500)
  const body = $('body').text().replace(/\s+/g, ' ').trim().slice(0, 5000)
  const lang = $('html').attr('lang') || 'en'
  const outlinks = []
  $('a[href]').each((_, el) => {
    try {
      const href = new URL($(el).attr('href'), url).href
      if (href.startsWith('http')) outlinks.push(href)
    } catch {}
  })
  const domain = new URL(url).hostname.replace(/^www\./, '')
  const id = domain.replace(/[^a-zA-Z0-9_-]/g, '_')
  return {
    id,
    url,
    domain,
    title: title || domain,
    description,
    body,
    lang,
    outlinks: [...new Set(outlinks)].slice(0, 50),
    crawledAt: new Date().toISOString(),
  }
 }
--- a/crawler/src/tranco.js
+++ b/crawler/src/tranco.js
@ -0,0 +1,44 @@
 import { createWriteStream, existsSync, readFileSync } from 'fs'
 import { pipeline } from 'stream/promises'
 // Majestic Million - stable URL, updated daily, rank,domain format
 const LIST_URL = 'https://downloads.majestic.com/majestic_million.csv'
 const LIST_FILE = './domains.csv'
 export async function downloadTranco() {
  if (existsSync(LIST_FILE)) {
    console.log('Domain list already downloaded')
    return
  }
  console.log('Downloading Majestic Million domain list...')
  const res = await fetch(LIST_URL)
  if (!res.ok) throw new Error(`Failed: ${res.status}`)
  await pipeline(res.body, createWriteStream(LIST_FILE))
  console.log('Download complete')
 }
 export function* readDomains(limit = Infinity) {
  const csv = readFileSync(LIST_FILE, 'utf8')
  let count = 0
  let header = true
  for (const line of csv.split('\n')) {
    if (!line.trim()) continue
    // Skip header row
    if (header) { header = false; continue }
    const cols = line.split(',')
    const rank = parseInt(cols[0])
    const domain = cols[2]?.trim()  // Majestic: GlobalRank,TldRank,Domain,...
    if (!domain || isNaN(rank)) continue
    yield { rank, domain }
    if (++count >= limit) break
  }
 }
 if (process.argv[1].endsWith('tranco.js')) {
  await downloadTranco()
  let i = 0
  for (const entry of readDomains(10)) {
    console.log(entry)
    i++
  }
 }
--- a/frontend/package-lock.json
+++ b/frontend/package-lock.json
--- a/frontend/package.json
+++ b/frontend/package.json
@ -0,0 +1,16 @@
 {
  "name": "search-engine-frontend",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "dev": "wrangler dev",
    "deploy": "wrangler deploy"
  },
  "dependencies": {
    "hono": "^4.7.0"
  },
  "devDependencies": {
    "wrangler": "^3.0.0",
    "@cloudflare/workers-types": "^4.0.0"
  }
 }
--- a/frontend/src/index.ts
+++ b/frontend/src/index.ts
@ -0,0 +1,239 @@
 const CSS = `
  @import url('https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@400;500;700&display=swap');
  *, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
  /* Dark (default) */
  :root {
    --bg: #0a0a0a; --surface: #111; --border: #222;
    --text: #e8e8e8; --muted: #666; --accent: #ff0099;
    --mark: rgba(255,0,153,0.2); --title: #fff;
    --footer-text: #333; --footer-link: #444; --snippet: #888; --peers: #444;
  }
  /* Light */
  html.light {
    --bg: #f4f4f4; --surface: #fff; --border: #ddd;
    --text: #111; --muted: #888; --accent: #cc007a;
    --mark: rgba(204,0,122,0.15); --title: #0a0a0a;
    --footer-text: #aaa; --footer-link: #999; --snippet: #555; --peers: #bbb;
  }
  body { background: var(--bg); color: var(--text); font-family: 'Roboto Mono', monospace; min-height: 100vh; transition: background .2s, color .2s; }
  a { color: var(--accent); text-decoration: none; }
  a:hover { filter: brightness(1.15); }
  mark { background: var(--mark); color: inherit; padding: 0 2px; }
  /* Theme toggle */
  .theme-toggle {
    position: fixed; top: 16px; right: 16px;
    background: none; border: 1px solid var(--border);
    font-size: 16px; padding: 6px 10px; cursor: pointer; line-height: 1;
    transition: border-color .15s; z-index: 100;
  }
  .theme-toggle:hover { border-color: var(--accent); }
  /* Home page */
  .home { display: flex; flex-direction: column; align-items: center; justify-content: center; min-height: 100vh; gap: 40px; }
  .home-brand { text-align: center; }
  .logo { font-size: 48px; font-weight: 700; letter-spacing: -1px; color: var(--title); line-height: 1; }
  .logo span { color: var(--accent); }
  .tagline { color: var(--muted); font-size: 12px; letter-spacing: 3px; text-transform: uppercase; margin-top: 10px; }
  /* Search bar */
  .search-form { display: flex; flex-direction: column; gap: 12px; width: 100%; max-width: 580px; padding: 0 16px; }
  .search-input {
    width: 100%; background: var(--surface); border: 1px solid var(--border);
    border-radius: 0; padding: 14px 18px; font-size: 15px; color: var(--text);
    font-family: 'Roboto Mono', monospace; outline: none; transition: border-color .15s, background .2s;
  }
  .search-input:focus { border-color: var(--accent); }
  .search-input::placeholder { color: var(--muted); }
  .search-btn {
    background: var(--accent); border: none; padding: 12px;
    color: #fff; font-size: 13px; font-weight: 700; font-family: 'Roboto Mono', monospace;
    letter-spacing: 2px; text-transform: uppercase; cursor: pointer; transition: filter .15s;
  }
  .search-btn:hover { filter: brightness(1.15); }
  /* Results page */
  .results-page { max-width: 720px; margin: 0 auto; padding: 0 16px 80px; }
  .results-header {
    display: flex; align-items: center; gap: 16px;
    padding: 20px 0; border-bottom: 1px solid var(--border); margin-bottom: 28px;
  }
  .results-logo { font-size: 20px; font-weight: 700; white-space: nowrap; color: var(--title); }
  .results-logo span { color: var(--accent); }
  .results-form { display: flex; gap: 8px; flex: 1; }
  .results-input {
    flex: 1; background: var(--surface); border: 1px solid var(--border);
    border-radius: 0; padding: 9px 14px; font-size: 14px; color: var(--text);
    font-family: 'Roboto Mono', monospace; outline: none; transition: border-color .15s, background .2s;
  }
  .results-input:focus { border-color: var(--accent); }
  .results-btn {
    background: var(--accent); border: none; padding: 9px 18px;
    color: #fff; font-size: 12px; font-weight: 700; font-family: 'Roboto Mono', monospace;
    letter-spacing: 1px; text-transform: uppercase; cursor: pointer; white-space: nowrap;
  }
  .results-btn:hover { filter: brightness(1.15); }
  .meta { color: var(--muted); font-size: 12px; margin-bottom: 28px; letter-spacing: 0.5px; }
  .result { margin-bottom: 32px; border-left: 2px solid transparent; padding-left: 16px; transition: border-color .15s; }
  .result:hover { border-left-color: var(--accent); }
  .result-url { font-size: 11px; color: var(--muted); margin-bottom: 5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; letter-spacing: 0.5px; }
  .result-title { font-size: 17px; font-weight: 500; line-height: 1.4; margin-bottom: 6px; }
  .result-title a { color: var(--title); }
  .result-title a:hover { color: var(--accent); }
  .result-snippet { font-size: 13px; color: var(--snippet); line-height: 1.7; }
  .pagination { display: flex; gap: 8px; margin-top: 40px; }
  .page-btn {
    background: var(--surface); border: 1px solid var(--border);
    padding: 8px 16px; color: var(--muted); font-size: 12px; font-family: 'Roboto Mono', monospace;
    letter-spacing: 1px; text-transform: uppercase; text-decoration: none; display: inline-block;
    transition: border-color .15s, color .15s;
  }
  .page-btn:hover { border-color: var(--accent); color: var(--accent); }
  .no-results { text-align: center; padding: 80px 20px; color: var(--muted); font-size: 13px; line-height: 2; }
  .peers { color: var(--peers); margin-left: 8px; }
  .footer { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--border); font-size: 11px; color: var(--footer-text); letter-spacing: 1px; }
  .footer a { color: var(--footer-link); }
  .footer a:hover { color: var(--accent); }
 `
 const THEME_JS = `
  (function() {
    var t = localStorage.getItem('theme');
    if (t === 'light') document.documentElement.classList.add('light');
  })();
  document.addEventListener('DOMContentLoaded', function() {
    var btn = document.getElementById('theme-toggle');
    function update() {
      var light = document.documentElement.classList.contains('light');
      btn.textContent = light ? '☀️' : '🌙';
    }
    update();
    btn.addEventListener('click', function() {
      document.documentElement.classList.toggle('light');
      localStorage.setItem('theme', document.documentElement.classList.contains('light') ? 'light' : 'dark');
      update();
    });
  });
 `
 function layout(title: string, body: string) {
  return `<!DOCTYPE html><html lang="en"><head>
    <meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1">
    <title>${title}</title>
    <link rel="preconnect" href="https://fonts.googleapis.com">
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
    <style>${CSS}</style>
    <script>${THEME_JS}</script>
  </head><body>
    <button class="theme-toggle" id="theme-toggle">🌙</button>
    ${body}
  </body></html>`
 }
 function homePage(q = '') {
  return layout('Radical Search', `
    <div class="home">
      <div class="home-brand">
        <div class="logo">RADICAL<span>_</span>SEARCH</div>
        <div class="tagline">The open web &mdash; unfiltered</div>
      </div>
      <form class="search-form" action="/search" method="get">
        <input class="search-input" name="q" type="search" placeholder="what are you looking for?" value="${esc(q)}" autofocus>
        <button class="search-btn" type="submit">Search the web</button>
      </form>
    </div>
  `)
 }
 function resultsPage(q: string, data: any, page: number) {
  const hits = data.hits ?? []
  const total = data.total ?? 0
  const peers = data.peers ?? []
  const results = hits.length
    ? hits.map((h: any) => {
        const title = h._formatted?.title || h.title || h.domain
        const snippet = h._formatted?.description || h.description || ''
        return `
          <div class="result">
            <div class="result-url">${esc(h.url)}</div>
            <div class="result-title"><a href="${esc(h.url)}" rel="noopener">${title}</a></div>
            ${snippet ? `<div class="result-snippet">${snippet}</div>` : ''}
          </div>`
      }).join('')
    : `<div class="no-results">no results for &ldquo;<strong style="color:var(--title)">${esc(q)}</strong>&rdquo;<br>the index is still growing &mdash; try again soon</div>`
  const prevLink = page > 0 ? `<a class="page-btn" href="/search?q=${encodeURIComponent(q)}&page=${page - 1}">&larr; prev</a>` : ''
  const nextLink = hits.length === 10 ? `<a class="page-btn" href="/search?q=${encodeURIComponent(q)}&page=${page + 1}">next &rarr;</a>` : ''
  return layout(`${esc(q)} — Radical Search`, `
    <div class="results-page">
      <div class="results-header">
        <a class="results-logo" href="/">RADICAL<span>_</span>SEARCH</a>
        <form class="results-form" action="/search" method="get">
          <input class="results-input" name="q" type="search" value="${esc(q)}" autofocus>
          <button class="results-btn" type="submit">Go</button>
        </form>
      </div>
      <div class="meta">
        ~${total.toLocaleString()} results
        ${peers.length ? `<span class="peers">// ${peers.length} peer${peers.length !== 1 ? 's' : ''} federated</span>` : ''}
      </div>
      ${results}
      ${hits.length ? `<div class="pagination">${prevLink}${nextLink}</div>` : ''}
      <div class="footer">
        <a href="https://theradicalparty.com">theradicalparty.com</a>
        &nbsp;&mdash;&nbsp;
        ${(data.nodeId ?? 'main')} node
      </div>
    </div>
  `)
 }
 function esc(s: string) {
  return String(s ?? '').replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;')
 }
 export default {
  async fetch(req: Request, env: { API_URL: string }) {
    const url = new URL(req.url)
    if (url.pathname === '/') {
      return new Response(homePage(), { headers: { 'Content-Type': 'text/html; charset=utf-8' } })
    }
    if (url.pathname === '/search') {
      const q = url.searchParams.get('q')?.trim() ?? ''
      if (!q) return Response.redirect(url.origin, 302)
      const page = Math.max(0, parseInt(url.searchParams.get('page') ?? '0'))
      try {
        const apiRes = await fetch(
          `${env.API_URL}/search?q=${encodeURIComponent(q)}&page=${page}`,
          { headers: { 'User-Agent': 'SearchFrontend/1.0' } }
        )
        const data = await apiRes.json() as any
        return new Response(resultsPage(q, data, page), {
          headers: { 'Content-Type': 'text/html; charset=utf-8' }
        })
      } catch (e: any) {
        return new Response(resultsPage(q, { hits: [], total: 0 }, page), {
          headers: { 'Content-Type': 'text/html; charset=utf-8' }
        })
      }
    }
    // Proxy API calls through (for stats, peers, etc.)
    if (url.pathname.startsWith('/api/')) {
      const apiPath = url.pathname.replace('/api', '')
      const apiRes = await fetch(`${env.API_URL}${apiPath}${url.search}`)
      return apiRes
    }
    return new Response('Not found', { status: 404 })
  }
 }
--- a/frontend/wrangler.jsonc
+++ b/frontend/wrangler.jsonc
@ -0,0 +1,12 @@
 {
  "name": "search-engine-frontend",
  "main": "src/index.ts",
  "compatibility_date": "2025-01-01",
  "compatibility_flags": ["nodejs_compat"],
  "routes": [
    { "pattern": "search.theradicalparty.com/*", "zone_name": "theradicalparty.com" }
  ],
  "vars": {
    "API_URL": "http://search-api.theradicalparty.com:3000"
  }
 }
--- a/scripts/deploy.env.example
+++ b/scripts/deploy.env.example
@ -0,0 +1,4 @@
 # Copy to deploy.env (git-ignored) and fill in. Used by deploy.sh.
 VM_HOST=your.vm.ip.address
 VM_USER=root
 VM_PASS=your-vm-password
--- a/scripts/deploy.sh
+++ b/scripts/deploy.sh
@ -0,0 +1,27 @@
 #!/bin/bash
 set -e
 # Load VM credentials from git-ignored deploy.env (copy deploy.env.example).
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 if [ -f "$SCRIPT_DIR/deploy.env" ]; then
  set -a; . "$SCRIPT_DIR/deploy.env"; set +a
 fi
 VM_HOST="${VM_HOST:?set VM_HOST in scripts/deploy.env}"
 VM_USER="${VM_USER:?set VM_USER in scripts/deploy.env}"
 VM_PASS="${VM_PASS:?set VM_PASS in scripts/deploy.env}"
 ssh_cmd() { sshpass -p "$VM_PASS" ssh -o StrictHostKeyChecking=no "$VM_USER@$VM_HOST" "$@"; }
 scp_cmd() { sshpass -p "$VM_PASS" scp -o StrictHostKeyChecking=no -r "$@"; }
 echo "=== Syncing files to VM ==="
 ssh_cmd "mkdir -p /tmp/search-api /tmp/search-crawler"
 scp_cmd ../api/. "$VM_USER@$VM_HOST:/tmp/search-api/"
 scp_cmd ../crawler/. "$VM_USER@$VM_HOST:/tmp/search-crawler/"
 scp_cmd install-vm.sh "$VM_USER@$VM_HOST:/tmp/install-vm.sh"
 echo "=== Running install script ==="
 ssh_cmd "bash /tmp/install-vm.sh"
 echo "=== Done ==="
 ssh_cmd "systemctl status meilisearch --no-pager && systemctl status search-api --no-pager"
--- a/scripts/install-vm.sh
+++ b/scripts/install-vm.sh
@ -0,0 +1,74 @@
 #!/bin/bash
 set -e
 echo "=== Installing Meilisearch ==="
 curl -L https://install.meilisearch.com | sh
 mv ./meilisearch /usr/local/bin/meilisearch
 echo "=== Creating Meilisearch systemd service ==="
 cat > /etc/systemd/system/meilisearch.service << 'EOF'
 [Unit]
 Description=Meilisearch Search Engine
 After=network.target
 [Service]
 User=root
 ExecStart=/usr/local/bin/meilisearch --http-addr 0.0.0.0:7700 --master-key masterKey --db-path /var/lib/meilisearch/data
 Restart=always
 RestartSec=5
 Environment=MEILI_NO_ANALYTICS=true
 [Install]
 WantedBy=multi-user.target
 EOF
 mkdir -p /var/lib/meilisearch/data
 systemctl daemon-reload
 systemctl enable meilisearch
 systemctl start meilisearch
 echo "Meilisearch started"
 echo "=== Deploying Search API ==="
 mkdir -p /opt/search-api
 cp -r /tmp/search-api/. /opt/search-api/
 cd /opt/search-api
 npm install --production
 cat > /etc/systemd/system/search-api.service << 'EOF'
 [Unit]
 Description=Search Engine API
 After=network.target meilisearch.service
 [Service]
 User=root
 WorkingDirectory=/opt/search-api
 ExecStart=/usr/bin/node src/index.js
 Restart=always
 RestartSec=5
 Environment=PORT=3000
 Environment=MEILI_URL=http://localhost:7700
 Environment=MEILI_KEY=masterKey
 Environment=NODE_ID=main
 [Install]
 WantedBy=multi-user.target
 EOF
 systemctl daemon-reload
 systemctl enable search-api
 systemctl start search-api
 echo "Search API started on :3000"
 echo "=== Deploying Crawler ==="
 mkdir -p /opt/search-crawler
 cp -r /tmp/search-crawler/. /opt/search-crawler/
 cd /opt/search-crawler
 npm install
 echo ""
 echo "Done! Services running:"
 echo "  Meilisearch: http://localhost:7700"
 echo "  Search API:  http://localhost:3000"
 echo ""
 echo "To start crawling:"
 echo "  cd /opt/search-crawler && LIMIT=50000 CONCURRENCY=8 npm start"