3 ms·
Does paywallskip scrape archive.is and archive.org?
by rendall 2y ago
Does paywallskip scrape archive.is and archive.org?
- francocanzani 2y agoIt fallbacks to archives, yes. Basically I use different User-Agent headers, different Referer headers, it tries disabling javascript once the page has loaded and the fallback is to fetch from web archives (Wayback Machine, archive.is, Google cache). Then the HTML is validated and parsed.