maverick/paywall
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
2 commits 1 branch 0 tags 54 KiB
Find a file
King Omar 665cd264cc Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 
Hard paywalls (AFR etc.) block Googlebot by IP but serve full articles to
facebookexternalhit/Twitterbot for link previews. fetchBest() now fetches
several crawler identities concurrently and keeps the copy with the most text,
so the Reader works on AFR (0 -> 2256 words). Also rank archives first for
known hard-paywall domains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 15:21:36 +10:00
src Multi-crawler fetch: try Googlebot + social crawlers, keep richest copy 2026-07-25 15:21:36 +10:00
.gitignore PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
package-lock.json PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
package.json PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00
wrangler.toml PaywallFree: read paywalled articles free — Radical reader + proxy + method hub 2026-07-25 15:02:18 +10:00