Initial commit: RADICAL_SEARCH self-hosted web search engine

Crawler (Majestic Million → undici → cheerio → Meilisearch), Hono search API with P2P peer federation, Cloudflare Worker frontend, VM provisioning scripts. Adds README, .gitignore, and moves VM credentials out of deploy.sh into a git-ignored deploy.env. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:52:31 +10:00 · 2026-07-22 01:52:31 +10:00 · b3aea18d9c
commit b3aea18d9c
17 changed files with 2471 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,20 @@
+# Dependencies
+node_modules/
+
+# Crawler data
+crawler/domains.csv
+crawler/*.csv
+
+# Wrangler / Cloudflare
+.wrangler/
+dist/
+
+# Local env / secrets (never commit VM passwords, keys)
+.env
+.env.local
+deploy.env
+*.local
+
+# Logs & OS
+*.log
+.DS_Store
--- a/README.md
+++ b/README.md
@ -0,0 +1,94 @@
+# 🔍 RADICAL_SEARCH
+
+> **The open web — unfiltered.** A self-hosted, from-scratch web search engine.
+
+RADICAL_SEARCH crawls the most-linked domains on the web, indexes them in
+[Meilisearch](https://www.meilisearch.com/), and serves results through a
+Cloudflare Worker frontend at **[search.theradicalparty.com](https://search.theradicalparty.com)**.
+The search API also supports optional **P2P federation** — nodes can register as
+peers and fan queries out across a cluster.
+
+## Architecture
+
+```
+                    ┌─────────────────────────────┐
+   browser  ───▶    │  frontend/  (Cloudflare Worker)  │  search.theradicalparty.com
+                    │  SSR UI, dark/light, proxies /search │
+                    └───────────────┬─────────────┘
+                                    │  fetch(API_URL)  (HTTPS :443)
+                                    ▼
+                    ┌─────────────────────────────┐
+                    │  api/  (Hono, Node, on VM)   │  search-api.theradicalparty.com
+                    │  /search /peers /stats /health │
+                    └───────────────┬─────────────┘
+                                    │  localhost:7700
+                                    ▼
+                    ┌─────────────────────────────┐
+                    │  Meilisearch  (index: pages) │
+                    └───────────────▲─────────────┘
+                                    │  batched docs
+                    ┌───────────────┴─────────────┐
+                    │  crawler/  (Node, on VM)     │  Majestic Million → fetch → parse → index
+                    └─────────────────────────────┘
+```
+
+## Components
+
+| Dir | What it is | Runtime |
+|-----|-----------|---------|
+| `crawler/` | Downloads the [Majestic Million](https://majestic.com/reports/majestic-million) domain list, crawls each domain (undici, 8s timeout, HTML-only, 500 KB cap), parses with cheerio (title/description/body/outlinks), and batches docs into Meilisearch. | Node on the VM |
+| `api/` | Hono search API. `/search` proxies to Meilisearch with highlighting and optional P2P peer fan-out. Also `/peers`, `/peers/register`, `/stats`, `/health`. | Node on the VM (`:PORT`) |
+| `frontend/` | Cloudflare Worker. Server-rendered UI (Roboto Mono, dark/light theme), proxies `/search` to `API_URL`. | Cloudflare Workers |
+| `scripts/` | `install-vm.sh` provisions Meilisearch + api + crawler as systemd services; `deploy.sh` syncs & runs it over SSH. | local → VM |
+
+## Local development
+
+```bash
+# 1. Meilisearch (Docker)
+docker run -it --rm -p 7700:7700 getmeili/meilisearch:latest --master-key masterKey
+
+# 2. API
+cd api && npm install && MEILI_KEY=masterKey npm run dev      # :3000
+
+# 3. A small crawl
+cd crawler && npm install && LIMIT=500 CONCURRENCY=8 npm start
+
+# 4. Frontend
+cd frontend && npm install && npm run dev                     # wrangler dev
+```
+
+## Deploy
+
+**Frontend (Cloudflare Worker):**
+```bash
+cd frontend && npm run deploy
+```
+
+**API + crawler (VM):** copy `scripts/deploy.env.example` → `scripts/deploy.env`,
+fill in the VM host/user/password (this file is git-ignored), then:
+```bash
+cd scripts && ./deploy.sh
+```
+
+### API exposure
+
+The frontend Worker reaches the API over **HTTPS on port 443** — a deployed
+Cloudflare Worker's `fetch()` cannot reliably use non-standard ports like 3000.
+The API is fronted at `search-api.theradicalparty.com` (see
+[`docs/DEPLOY.md`](docs/DEPLOY.md) for the port/DNS setup).
+
+## Configuration
+
+| Var | Component | Default | Notes |
+|-----|-----------|---------|-------|
+| `MEILI_URL` | api, crawler | `http://localhost:7700` | Meilisearch endpoint |
+| `MEILI_KEY` | api, crawler | `masterKey` | Meilisearch master key (VM-local only) |
+| `PORT` | api | `3000` | API listen port |
+| `NODE_ID` | api | `main` | P2P node identity |
+| `CONCURRENCY` | crawler | `8` | Parallel fetches |
+| `LIMIT` | crawler | `100000` | Max domains to crawl |
+| `API_URL` | frontend | — | Set in `frontend/wrangler.jsonc` `vars` |
+
+## License
+
+Private / unlicensed — © the maintainer.
--- a/api/package.json
+++ b/api/package.json
@ -0,0 +1,13 @@
+{
+  "name": "search-engine-api",
+  "version": "1.0.0",
+  "type": "module",
+  "scripts": {
+    "start": "node src/index.js",
+    "dev": "node --watch src/index.js"
+  },
+  "dependencies": {
+    "hono": "^4.7.0",
+    "@hono/node-server": "^1.14.0"
+  }
+}
--- a/api/src/index.js
+++ b/api/src/index.js
@ -0,0 +1,115 @@
+import { Hono } from 'hono'
+import { serve } from '@hono/node-server'
+
+const app = new Hono()
+const PORT = parseInt(process.env.PORT || '3000')
+const MEILI_URL = process.env.MEILI_URL || 'http://localhost:7700'
+const MEILI_KEY = process.env.MEILI_KEY || 'masterKey'
+const NODE_ID = process.env.NODE_ID || 'main'
+
+// In-memory peer registry  { id, url, lastSeen }
+const peers = new Map()
+
+async function meiliSearch(q, offset = 0, limit = 10) {
+  const res = await fetch(`${MEILI_URL}/indexes/pages/search`, {
+    method: 'POST',
+    headers: {
+      'Authorization': `Bearer ${MEILI_KEY}`,
+      'Content-Type': 'application/json',
+    },
+    body: JSON.stringify({
+      q,
+      limit,
+      offset,
+      attributesToHighlight: ['title', 'description'],
+      highlightPreTag: '<mark>',
+      highlightPostTag: '</mark>',
+      attributesToRetrieve: ['url', 'domain', 'title', 'description', 'crawledAt'],
+    }),
+  })
+  return res.json()
+}
+
+async function queryPeer(peer, q) {
+  try {
+    const res = await fetch(`${peer.url}/search?q=${encodeURIComponent(q)}&local=1`, {
+      signal: AbortSignal.timeout(3000),
+      headers: { 'X-Peer-Id': NODE_ID },
+    })
+    const data = await res.json()
+    return data.hits ?? []
+  } catch {
+    return []
+  }
+}
+
+// Search endpoint — fans out to peers when not local
+app.get('/search', async (c) => {
+  const q = c.req.query('q')?.trim()
+  const page = Math.max(0, parseInt(c.req.query('page') ?? '0'))
+  const local = c.req.query('local') === '1'
+
+  if (!q) return c.json({ error: 'missing q' }, 400)
+
+  const offset = page * 10
+  const local$ = meiliSearch(q, offset)
+  const peer$ = local ? [] : [...peers.values()].map(p => queryPeer(p, q))
+
+  const [localResult, ...peerResults] = await Promise.all([local$, ...peer$])
+
+  // Deduplicate and merge peer hits
+  const seen = new Set(localResult.hits?.map(h => h.url) ?? [])
+  const peerHits = peerResults
+    .flat()
+    .filter(h => !seen.has(h.url))
+    .slice(0, 5)
+
+  const hits = [...(localResult.hits ?? []), ...peerHits]
+
+  return c.json({
+    hits,
+    total: (localResult.estimatedTotalHits ?? 0) + peerHits.length,
+    page,
+    query: q,
+    nodeId: NODE_ID,
+    peers: [...peers.keys()],
+  })
+})
+
+// P2P: register as peer
+app.post('/peers/register', async (c) => {
+  const { id, url } = await c.req.json()
+  if (!id || !url) return c.json({ error: 'id and url required' }, 400)
+  peers.set(id, { id, url, lastSeen: Date.now() })
+  return c.json({ ok: true, peerId: NODE_ID, peers: [...peers.keys()] })
+})
+
+// P2P: list peers
+app.get('/peers', (c) => {
+  return c.json({ nodeId: NODE_ID, peers: [...peers.values()] })
+})
+
+// Health
+app.get('/health', async (c) => {
+  try {
+    const res = await fetch(`${MEILI_URL}/health`, {
+      headers: { 'Authorization': `Bearer ${MEILI_KEY}` }
+    })
+    const meili = await res.json()
+    return c.json({ ok: true, meili, nodeId: NODE_ID })
+  } catch (e) {
+    return c.json({ ok: false, error: e.message }, 503)
+  }
+})
+
+// Stats
+app.get('/stats', async (c) => {
+  const res = await fetch(`${MEILI_URL}/indexes/pages/stats`, {
+    headers: { 'Authorization': `Bearer ${MEILI_KEY}` }
+  })
+  const stats = await res.json()
+  return c.json({ ...stats, peers: peers.size, nodeId: NODE_ID })
+})
+
+serve({ fetch: app.fetch, port: PORT })
+console.log(`Search API running on :${PORT} [node=${NODE_ID}]`)
--- a/crawler/package.json
+++ b/crawler/package.json
@ -0,0 +1,14 @@
+{
+  "name": "search-engine-crawler",
+  "version": "1.0.0",
+  "type": "module",
+  "scripts": {
+    "start": "node src/index.js",
+    "tranco": "node src/tranco.js"
+  },
+  "dependencies": {
+    "cheerio": "^1.0.0",
+    "p-limit": "^6.2.0",
+    "undici": "^7.0.0"
+  }
+}
--- a/crawler/src/fetcher.js
+++ b/crawler/src/fetcher.js
@ -0,0 +1,47 @@
+import { fetch } from 'undici'
+
+const TIMEOUT_MS = 8000
+const MAX_BODY_BYTES = 500_000
+
+const HEADERS = {
+  'User-Agent': 'SearchBot/1.0 (compatible; +https://search.theradicalparty.com/bot)',
+  'Accept': 'text/html,application/xhtml+xml',
+  'Accept-Language': 'en',
+}
+
+export async function fetchPage(domain) {
+  const urls = [`https://${domain}`, `http://${domain}`]
+  for (const url of urls) {
+    try {
+      const controller = new AbortController()
+      const timer = setTimeout(() => controller.abort(), TIMEOUT_MS)
+      const res = await fetch(url, {
+        headers: HEADERS,
+        signal: controller.signal,
+        maxRedirections: 3,
+      })
+      clearTimeout(timer)
+
+      if (!res.ok) continue
+      const contentType = res.headers.get('content-type') ?? ''
+      if (!contentType.includes('html')) continue
+
+      // Read up to MAX_BODY_BYTES
+      const reader = res.body.getReader()
+      const chunks = []
+      let total = 0
+      while (true) {
+        const { done, value } = await reader.read()
+        if (done) break
+        chunks.push(value)
+        total += value.length
+        if (total >= MAX_BODY_BYTES) { reader.cancel(); break }
+      }
+      const html = Buffer.concat(chunks).toString('utf8')
+      return { url, html, status: res.status }
+    } catch {
+      // try next URL
+    }
+  }
+  return null
+}
--- a/crawler/src/index.js
+++ b/crawler/src/index.js
@ -0,0 +1,66 @@
+import pLimit from 'p-limit'
+import { downloadTranco, readDomains } from './tranco.js'
+import { fetchPage } from './fetcher.js'
+import { parsePage } from './parser.js'
+import { setupIndex, indexDoc, flush } from './indexer.js'
+
+const CONCURRENCY = parseInt(process.env.CONCURRENCY || '8')
+const LIMIT = parseInt(process.env.LIMIT || '100000')
+
+let crawled = 0
+let failed = 0
+let skipped = 0
+
+function log() {
+  if (crawled % 50 === 0) {
+    process.stdout.write(`\r[crawled=${crawled} failed=${failed} skipped=${skipped}]`)
+  }
+}
+
+async function crawlDomain({ rank, domain }) {
+  const result = await fetchPage(domain)
+  if (!result) { failed++; log(); return }
+
+  const doc = parsePage(result.url, result.html)
+  if (!doc.title && !doc.description) { skipped++; log(); return }
+
+  doc.rank = rank
+  await indexDoc(doc)
+  crawled++
+  log()
+}
+
+// Process domains in a bounded sliding window instead of creating all promises upfront
+async function main() {
+  await downloadTranco()
+  await setupIndex()
+
+  console.log(`Starting crawl: concurrency=${CONCURRENCY} limit=${LIMIT}`)
+
+  const queue = []
+  let active = 0
+
+  async function runNext(entry) {
+    active++
+    try { await crawlDomain(entry) } catch {}
+    active--
+  }
+
+  for (const entry of readDomains(LIMIT)) {
+    while (active >= CONCURRENCY) {
+      await new Promise(r => setTimeout(r, 10))
+    }
+    queue.push(runNext(entry))
+    // Periodically drain settled promises to free memory
+    if (queue.length >= 500) {
+      await Promise.allSettled(queue.splice(0, 200))
+    }
+  }
+
+  await Promise.allSettled(queue)
+  await flush()
+
+  console.log(`\nDone. crawled=${crawled} failed=${failed} skipped=${skipped}`)
+}
+
+main().catch(console.error)
--- a/crawler/src/indexer.js
+++ b/crawler/src/indexer.js
@ -0,0 +1,40 @@
+const MEILI_URL = process.env.MEILI_URL || 'http://localhost:7700'
+const MEILI_KEY = process.env.MEILI_KEY || 'masterKey'
+const INDEX = 'pages'
+const BATCH_SIZE = 100
+
+let buffer = []
+
+async function meili(method, path, body) {
+  const res = await fetch(`${MEILI_URL}${path}`, {
+    method,
+    headers: {
+      'Authorization': `Bearer ${MEILI_KEY}`,
+      'Content-Type': 'application/json',
+    },
+    body: body ? JSON.stringify(body) : undefined,
+  })
+  return res.json()
+}
+
+export async function setupIndex() {
+  await meili('POST', '/indexes', { uid: INDEX, primaryKey: 'id' })
+  await meili('PATCH', `/indexes/${INDEX}/settings`, {
+    searchableAttributes: ['title', 'description', 'domain', 'body'],
+    displayedAttributes: ['id', 'url', 'domain', 'title', 'description', 'crawledAt'],
+    rankingRules: ['words', 'typo', 'proximity', 'attribute', 'sort', 'exactness'],
+    filterableAttributes: ['lang', 'domain'],
+  })
+  console.log('Index ready')
+}
+
+export async function indexDoc(doc) {
+  buffer.push(doc)
+  if (buffer.length >= BATCH_SIZE) await flush()
+}
+
+export async function flush() {
+  if (buffer.length === 0) return
+  const docs = buffer.splice(0)
+  await meili('POST', `/indexes/${INDEX}/documents`, docs)
+}
--- a/crawler/src/parser.js
+++ b/crawler/src/parser.js
@ -0,0 +1,43 @@
+import { load } from 'cheerio'
+
+export function parsePage(url, html) {
+  const $ = load(html)
+
+  // Remove noise
+  $('script, style, noscript, nav, footer, header, aside, [aria-hidden=true]').remove()
+
+  const title = ($('title').text() || $('h1').first().text()).trim().slice(0, 200)
+
+  const description = (
+    $('meta[name=description]').attr('content') ||
+    $('meta[property="og:description"]').attr('content') ||
+    $('p').first().text()
+  ).trim().slice(0, 500)
+
+  const body = $('body').text().replace(/\s+/g, ' ').trim().slice(0, 5000)
+
+  const lang = $('html').attr('lang') || 'en'
+
+  const outlinks = []
+  $('a[href]').each((_, el) => {
+    try {
+      const href = new URL($(el).attr('href'), url).href
+      if (href.startsWith('http')) outlinks.push(href)
+    } catch {}
+  })
+
+  const domain = new URL(url).hostname.replace(/^www\./, '')
+  const id = domain.replace(/[^a-zA-Z0-9_-]/g, '_')
+
+  return {
+    id,
+    url,
+    domain,
+    title: title || domain,
+    description,
+    body,
+    lang,
+    outlinks: [...new Set(outlinks)].slice(0, 50),
+    crawledAt: new Date().toISOString(),
+  }
+}
--- a/crawler/src/tranco.js
+++ b/crawler/src/tranco.js
@ -0,0 +1,44 @@
+import { createWriteStream, existsSync, readFileSync } from 'fs'
+import { pipeline } from 'stream/promises'
+
+// Majestic Million - stable URL, updated daily, rank,domain format
+const LIST_URL = 'https://downloads.majestic.com/majestic_million.csv'
+const LIST_FILE = './domains.csv'
+
+export async function downloadTranco() {
+  if (existsSync(LIST_FILE)) {
+    console.log('Domain list already downloaded')
+    return
+  }
+  console.log('Downloading Majestic Million domain list...')
+  const res = await fetch(LIST_URL)
+  if (!res.ok) throw new Error(`Failed: ${res.status}`)
+  await pipeline(res.body, createWriteStream(LIST_FILE))
+  console.log('Download complete')
+}
+
+export function* readDomains(limit = Infinity) {
+  const csv = readFileSync(LIST_FILE, 'utf8')
+  let count = 0
+  let header = true
+  for (const line of csv.split('\n')) {
+    if (!line.trim()) continue
+    // Skip header row
+    if (header) { header = false; continue }
+    const cols = line.split(',')
+    const rank = parseInt(cols[0])
+    const domain = cols[2]?.trim()  // Majestic: GlobalRank,TldRank,Domain,...
+    if (!domain || isNaN(rank)) continue
+    yield { rank, domain }
+    if (++count >= limit) break
+  }
+}
+
+if (process.argv[1].endsWith('tranco.js')) {
+  await downloadTranco()
+  let i = 0
+  for (const entry of readDomains(10)) {
+    console.log(entry)
+    i++
+  }
+}
--- a/frontend/package-lock.json
+++ b/frontend/package-lock.json
--- a/frontend/package.json
+++ b/frontend/package.json
@ -0,0 +1,16 @@
+{
+  "name": "search-engine-frontend",
+  "version": "1.0.0",
+  "private": true,
+  "scripts": {
+    "dev": "wrangler dev",
+    "deploy": "wrangler deploy"
+  },
+  "dependencies": {
+    "hono": "^4.7.0"
+  },
+  "devDependencies": {
+    "wrangler": "^3.0.0",
+    "@cloudflare/workers-types": "^4.0.0"
+  }
+}
--- a/frontend/src/index.ts
+++ b/frontend/src/index.ts
@ -0,0 +1,239 @@
+const CSS = `
+  @import url('https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@400;500;700&display=swap');
+  *, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
+
+  /* Dark (default) */
+  :root {
+    --bg: #0a0a0a; --surface: #111; --border: #222;
+    --text: #e8e8e8; --muted: #666; --accent: #ff0099;
+    --mark: rgba(255,0,153,0.2); --title: #fff;
+    --footer-text: #333; --footer-link: #444; --snippet: #888; --peers: #444;
+  }
+
+  /* Light */
+  html.light {
+    --bg: #f4f4f4; --surface: #fff; --border: #ddd;
+    --text: #111; --muted: #888; --accent: #cc007a;
+    --mark: rgba(204,0,122,0.15); --title: #0a0a0a;
+    --footer-text: #aaa; --footer-link: #999; --snippet: #555; --peers: #bbb;
+  }
+
+  body { background: var(--bg); color: var(--text); font-family: 'Roboto Mono', monospace; min-height: 100vh; transition: background .2s, color .2s; }
+  a { color: var(--accent); text-decoration: none; }
+  a:hover { filter: brightness(1.15); }
+  mark { background: var(--mark); color: inherit; padding: 0 2px; }
+
+  /* Theme toggle */
+  .theme-toggle {
+    position: fixed; top: 16px; right: 16px;
+    background: none; border: 1px solid var(--border);
+    font-size: 16px; padding: 6px 10px; cursor: pointer; line-height: 1;
+    transition: border-color .15s; z-index: 100;
+  }
+  .theme-toggle:hover { border-color: var(--accent); }
+
+  /* Home page */
+  .home { display: flex; flex-direction: column; align-items: center; justify-content: center; min-height: 100vh; gap: 40px; }
+  .home-brand { text-align: center; }
+  .logo { font-size: 48px; font-weight: 700; letter-spacing: -1px; color: var(--title); line-height: 1; }
+  .logo span { color: var(--accent); }
+  .tagline { color: var(--muted); font-size: 12px; letter-spacing: 3px; text-transform: uppercase; margin-top: 10px; }
+
+  /* Search bar */
+  .search-form { display: flex; flex-direction: column; gap: 12px; width: 100%; max-width: 580px; padding: 0 16px; }
+  .search-input {
+    width: 100%; background: var(--surface); border: 1px solid var(--border);
+    border-radius: 0; padding: 14px 18px; font-size: 15px; color: var(--text);
+    font-family: 'Roboto Mono', monospace; outline: none; transition: border-color .15s, background .2s;
+  }
+  .search-input:focus { border-color: var(--accent); }
+  .search-input::placeholder { color: var(--muted); }
+  .search-btn {
+    background: var(--accent); border: none; padding: 12px;
+    color: #fff; font-size: 13px; font-weight: 700; font-family: 'Roboto Mono', monospace;
+    letter-spacing: 2px; text-transform: uppercase; cursor: pointer; transition: filter .15s;
+  }
+  .search-btn:hover { filter: brightness(1.15); }
+
+  /* Results page */
+  .results-page { max-width: 720px; margin: 0 auto; padding: 0 16px 80px; }
+  .results-header {
+    display: flex; align-items: center; gap: 16px;
+    padding: 20px 0; border-bottom: 1px solid var(--border); margin-bottom: 28px;
+  }
+  .results-logo { font-size: 20px; font-weight: 700; white-space: nowrap; color: var(--title); }
+  .results-logo span { color: var(--accent); }
+  .results-form { display: flex; gap: 8px; flex: 1; }
+  .results-input {
+    flex: 1; background: var(--surface); border: 1px solid var(--border);
+    border-radius: 0; padding: 9px 14px; font-size: 14px; color: var(--text);
+    font-family: 'Roboto Mono', monospace; outline: none; transition: border-color .15s, background .2s;
+  }
+  .results-input:focus { border-color: var(--accent); }
+  .results-btn {
+    background: var(--accent); border: none; padding: 9px 18px;
+    color: #fff; font-size: 12px; font-weight: 700; font-family: 'Roboto Mono', monospace;
+    letter-spacing: 1px; text-transform: uppercase; cursor: pointer; white-space: nowrap;
+  }
+  .results-btn:hover { filter: brightness(1.15); }
+  .meta { color: var(--muted); font-size: 12px; margin-bottom: 28px; letter-spacing: 0.5px; }
+  .result { margin-bottom: 32px; border-left: 2px solid transparent; padding-left: 16px; transition: border-color .15s; }
+  .result:hover { border-left-color: var(--accent); }
+  .result-url { font-size: 11px; color: var(--muted); margin-bottom: 5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; letter-spacing: 0.5px; }
+  .result-title { font-size: 17px; font-weight: 500; line-height: 1.4; margin-bottom: 6px; }
+  .result-title a { color: var(--title); }
+  .result-title a:hover { color: var(--accent); }
+  .result-snippet { font-size: 13px; color: var(--snippet); line-height: 1.7; }
+  .pagination { display: flex; gap: 8px; margin-top: 40px; }
+  .page-btn {
+    background: var(--surface); border: 1px solid var(--border);
+    padding: 8px 16px; color: var(--muted); font-size: 12px; font-family: 'Roboto Mono', monospace;
+    letter-spacing: 1px; text-transform: uppercase; text-decoration: none; display: inline-block;
+    transition: border-color .15s, color .15s;
+  }
+  .page-btn:hover { border-color: var(--accent); color: var(--accent); }
+  .no-results { text-align: center; padding: 80px 20px; color: var(--muted); font-size: 13px; line-height: 2; }
+  .peers { color: var(--peers); margin-left: 8px; }
+  .footer { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--border); font-size: 11px; color: var(--footer-text); letter-spacing: 1px; }
+  .footer a { color: var(--footer-link); }
+  .footer a:hover { color: var(--accent); }
+`
+
+const THEME_JS = `
+  (function() {
+    var t = localStorage.getItem('theme');
+    if (t === 'light') document.documentElement.classList.add('light');
+  })();
+  document.addEventListener('DOMContentLoaded', function() {
+    var btn = document.getElementById('theme-toggle');
+    function update() {
+      var light = document.documentElement.classList.contains('light');
+      btn.textContent = light ? '☀️' : '🌙';
+    }
+    update();
+    btn.addEventListener('click', function() {
+      document.documentElement.classList.toggle('light');
+      localStorage.setItem('theme', document.documentElement.classList.contains('light') ? 'light' : 'dark');
+      update();
+    });
+  });
+`
+
+function layout(title: string, body: string) {
+  return `<!DOCTYPE html><html lang="en"><head>
+    <meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>${title}</title>
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <style>${CSS}</style>
+    <script>${THEME_JS}</script>
+  </head><body>
+    <button class="theme-toggle" id="theme-toggle">🌙</button>
+    ${body}
+  </body></html>`
+}
+
+function homePage(q = '') {
+  return layout('Radical Search', `
+    <div class="home">
+      <div class="home-brand">
+        <div class="logo">RADICAL<span>_</span>SEARCH</div>
+        <div class="tagline">The open web &mdash; unfiltered</div>
+      </div>
+      <form class="search-form" action="/search" method="get">
+        <input class="search-input" name="q" type="search" placeholder="what are you looking for?" value="${esc(q)}" autofocus>
+        <button class="search-btn" type="submit">Search the web</button>
+      </form>
+    </div>
+  `)
+}
+
+function resultsPage(q: string, data: any, page: number) {
+  const hits = data.hits ?? []
+  const total = data.total ?? 0
+  const peers = data.peers ?? []
+
+  const results = hits.length
+    ? hits.map((h: any) => {
+        const title = h._formatted?.title || h.title || h.domain
+        const snippet = h._formatted?.description || h.description || ''
+        return `
+          <div class="result">
+            <div class="result-url">${esc(h.url)}</div>
+            <div class="result-title"><a href="${esc(h.url)}" rel="noopener">${title}</a></div>
+            ${snippet ? `<div class="result-snippet">${snippet}</div>` : ''}
+          </div>`
+      }).join('')
+    : `<div class="no-results">no results for &ldquo;<strong style="color:var(--title)">${esc(q)}</strong>&rdquo;<br>the index is still growing &mdash; try again soon</div>`
+
+  const prevLink = page > 0 ? `<a class="page-btn" href="/search?q=${encodeURIComponent(q)}&page=${page - 1}">&larr; prev</a>` : ''
+  const nextLink = hits.length === 10 ? `<a class="page-btn" href="/search?q=${encodeURIComponent(q)}&page=${page + 1}">next &rarr;</a>` : ''
+
+  return layout(`${esc(q)} — Radical Search`, `
+    <div class="results-page">
+      <div class="results-header">
+        <a class="results-logo" href="/">RADICAL<span>_</span>SEARCH</a>
+        <form class="results-form" action="/search" method="get">
+          <input class="results-input" name="q" type="search" value="${esc(q)}" autofocus>
+          <button class="results-btn" type="submit">Go</button>
+        </form>
+      </div>
+      <div class="meta">
+        ~${total.toLocaleString()} results
+        ${peers.length ? `<span class="peers">// ${peers.length} peer${peers.length !== 1 ? 's' : ''} federated</span>` : ''}
+      </div>
+      ${results}
+      ${hits.length ? `<div class="pagination">${prevLink}${nextLink}</div>` : ''}
+      <div class="footer">
+        <a href="https://theradicalparty.com">theradicalparty.com</a>
+        &nbsp;&mdash;&nbsp;
+        ${(data.nodeId ?? 'main')} node
+      </div>
+    </div>
+  `)
+}
+
+function esc(s: string) {
+  return String(s ?? '').replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;')
+}
+
+export default {
+  async fetch(req: Request, env: { API_URL: string }) {
+    const url = new URL(req.url)
+
+    if (url.pathname === '/') {
+      return new Response(homePage(), { headers: { 'Content-Type': 'text/html; charset=utf-8' } })
+    }
+
+    if (url.pathname === '/search') {
+      const q = url.searchParams.get('q')?.trim() ?? ''
+      if (!q) return Response.redirect(url.origin, 302)
+
+      const page = Math.max(0, parseInt(url.searchParams.get('page') ?? '0'))
+
+      try {
+        const apiRes = await fetch(
+          `${env.API_URL}/search?q=${encodeURIComponent(q)}&page=${page}`,
+          { headers: { 'User-Agent': 'SearchFrontend/1.0' } }
+        )
+        const data = await apiRes.json() as any
+        return new Response(resultsPage(q, data, page), {
+          headers: { 'Content-Type': 'text/html; charset=utf-8' }
+        })
+      } catch (e: any) {
+        return new Response(resultsPage(q, { hits: [], total: 0 }, page), {
+          headers: { 'Content-Type': 'text/html; charset=utf-8' }
+        })
+      }
+    }
+
+    // Proxy API calls through (for stats, peers, etc.)
+    if (url.pathname.startsWith('/api/')) {
+      const apiPath = url.pathname.replace('/api', '')
+      const apiRes = await fetch(`${env.API_URL}${apiPath}${url.search}`)
+      return apiRes
+    }
+
+    return new Response('Not found', { status: 404 })
+  }
+}
--- a/frontend/wrangler.jsonc
+++ b/frontend/wrangler.jsonc
@ -0,0 +1,12 @@
+{
+  "name": "search-engine-frontend",
+  "main": "src/index.ts",
+  "compatibility_date": "2025-01-01",
+  "compatibility_flags": ["nodejs_compat"],
+  "routes": [
+    { "pattern": "search.theradicalparty.com/*", "zone_name": "theradicalparty.com" }
+  ],
+  "vars": {
+    "API_URL": "http://search-api.theradicalparty.com:3000"
+  }
+}
--- a/scripts/deploy.env.example
+++ b/scripts/deploy.env.example
@ -0,0 +1,4 @@
+# Copy to deploy.env (git-ignored) and fill in. Used by deploy.sh.
+VM_HOST=your.vm.ip.address
+VM_USER=root
+VM_PASS=your-vm-password
--- a/scripts/deploy.sh
+++ b/scripts/deploy.sh
@ -0,0 +1,27 @@
+#!/bin/bash
+set -e
+
+# Load VM credentials from git-ignored deploy.env (copy deploy.env.example).
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+if [ -f "$SCRIPT_DIR/deploy.env" ]; then
+  set -a; . "$SCRIPT_DIR/deploy.env"; set +a
+fi
+
+VM_HOST="${VM_HOST:?set VM_HOST in scripts/deploy.env}"
+VM_USER="${VM_USER:?set VM_USER in scripts/deploy.env}"
+VM_PASS="${VM_PASS:?set VM_PASS in scripts/deploy.env}"
+
+ssh_cmd() { sshpass -p "$VM_PASS" ssh -o StrictHostKeyChecking=no "$VM_USER@$VM_HOST" "$@"; }
+scp_cmd() { sshpass -p "$VM_PASS" scp -o StrictHostKeyChecking=no -r "$@"; }
+
+echo "=== Syncing files to VM ==="
+ssh_cmd "mkdir -p /tmp/search-api /tmp/search-crawler"
+scp_cmd ../api/. "$VM_USER@$VM_HOST:/tmp/search-api/"
+scp_cmd ../crawler/. "$VM_USER@$VM_HOST:/tmp/search-crawler/"
+scp_cmd install-vm.sh "$VM_USER@$VM_HOST:/tmp/install-vm.sh"
+
+echo "=== Running install script ==="
+ssh_cmd "bash /tmp/install-vm.sh"
+
+echo "=== Done ==="
+ssh_cmd "systemctl status meilisearch --no-pager && systemctl status search-api --no-pager"
--- a/scripts/install-vm.sh
+++ b/scripts/install-vm.sh
@ -0,0 +1,74 @@
+#!/bin/bash
+set -e
+
+echo "=== Installing Meilisearch ==="
+curl -L https://install.meilisearch.com | sh
+mv ./meilisearch /usr/local/bin/meilisearch
+
+echo "=== Creating Meilisearch systemd service ==="
+cat > /etc/systemd/system/meilisearch.service << 'EOF'
+[Unit]
+Description=Meilisearch Search Engine
+After=network.target
+
+[Service]
+User=root
+ExecStart=/usr/local/bin/meilisearch --http-addr 0.0.0.0:7700 --master-key masterKey --db-path /var/lib/meilisearch/data
+Restart=always
+RestartSec=5
+Environment=MEILI_NO_ANALYTICS=true
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+mkdir -p /var/lib/meilisearch/data
+systemctl daemon-reload
+systemctl enable meilisearch
+systemctl start meilisearch
+echo "Meilisearch started"
+
+echo "=== Deploying Search API ==="
+mkdir -p /opt/search-api
+cp -r /tmp/search-api/. /opt/search-api/
+cd /opt/search-api
+npm install --production
+
+cat > /etc/systemd/system/search-api.service << 'EOF'
+[Unit]
+Description=Search Engine API
+After=network.target meilisearch.service
+
+[Service]
+User=root
+WorkingDirectory=/opt/search-api
+ExecStart=/usr/bin/node src/index.js
+Restart=always
+RestartSec=5
+Environment=PORT=3000
+Environment=MEILI_URL=http://localhost:7700
+Environment=MEILI_KEY=masterKey
+Environment=NODE_ID=main
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl daemon-reload
+systemctl enable search-api
+systemctl start search-api
+echo "Search API started on :3000"
+
+echo "=== Deploying Crawler ==="
+mkdir -p /opt/search-crawler
+cp -r /tmp/search-crawler/. /opt/search-crawler/
+cd /opt/search-crawler
+npm install
+
+echo ""
+echo "Done! Services running:"
+echo "  Meilisearch: http://localhost:7700"
+echo "  Search API:  http://localhost:3000"
+echo ""
+echo "To start crawling:"
+echo "  cd /opt/search-crawler && LIMIT=50000 CONCURRENCY=8 npm start"