maverick/paywall
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
paywall/src
History
King Omar 665cd264cc Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 
Hard paywalls (AFR etc.) block Googlebot by IP but serve full articles to
facebookexternalhit/Twitterbot for link previews. fetchBest() now fetches
several crawler identities concurrently and keeps the copy with the most text,
so the Reader works on AFR (0 -> 2256 words). Also rank archives first for
known hard-paywall domains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 15:21:36 +10:00
..
index.js Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 2026-07-25 15:21:36 +10:00
methods.js Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 2026-07-25 15:21:36 +10:00
proxy.js Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 2026-07-25 15:21:36 +10:00
views.js PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00