A live 404/paywall stub with >80 words (e.g. a site's '404 - Page Not Found'
page) was being rendered as the article. fetchBest now ignores error-status
bodies and returns null when every crawler errors, so /reader correctly falls
through to the Wayback snapshot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the multi-crawler fetch can't get article text (FT, WSJ, etc. return
403/JS-only shells to every crawler), fall back to the Internet Archive:
resolve the closest snapshot via the availability API, fetch the raw id_
version, and extract THAT into the Radical Reader with a 'via Wayback Machine,
archived DATE' source note. archive.today can't be used server-side (429 +
captcha on datacenter IPs) so it stays as the user-side button.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hard paywalls (AFR etc.) block Googlebot by IP but serve full articles to
facebookexternalhit/Twitterbot for link previews. fetchBest() now fetches
several crawler identities concurrently and keeps the copy with the most text,
so the Reader works on AFR (0 -> 2256 words). Also rank archives first for
known hard-paywall domains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
