When the multi-crawler fetch can't get article text (FT, WSJ, etc. return
403/JS-only shells to every crawler), fall back to the Internet Archive:
resolve the closest snapshot via the availability API, fetch the raw id_
version, and extract THAT into the Radical Reader with a 'via Wayback Machine,
archived DATE' source note. archive.today can't be used server-side (429 +
captcha on datacenter IPs) so it stays as the user-side button.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hard paywalls (AFR etc.) block Googlebot by IP but serve full articles to
facebookexternalhit/Twitterbot for link previews. fetchBest() now fetches
several crawler identities concurrently and keeps the copy with the most text,
so the Reader works on AFR (0 -> 2256 words). Also rank archives first for
known hard-paywall domains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
