No description
Find a file
King Omar 88038a7c0e Wayback fallback: when live site blocks all crawlers, read from a Wayback snapshot
When the multi-crawler fetch can't get article text (FT, WSJ, etc. return
403/JS-only shells to every crawler), fall back to the Internet Archive:
resolve the closest snapshot via the availability API, fetch the raw id_
version, and extract THAT into the Radical Reader with a 'via Wayback Machine,
archived DATE' source note. archive.today can't be used server-side (429 +
captcha on datacenter IPs) so it stays as the user-side button.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 15:54:47 +10:00
src Wayback fallback: when live site blocks all crawlers, read from a Wayback snapshot 2026-07-25 15:54:47 +10:00
.gitignore PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
package-lock.json PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
package.json PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
wrangler.toml PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00