4 ms·
Cloudflare has more or less killed white-hat crawling en large. On top, you have sites like Bloomberg where even solutions like Puppeteer-stealth will fail at s
by marban 4y ago
Cloudflare has more or less killed white-hat crawling en large. On top, you have sites like Bloomberg where even solutions like Puppeteer-stealth will fail at some point.
- xwolfi 4y agoAnd Selenium with clever clicking ? I never got why crawlers never took the approach to just be a slow human clicking around chrome.
- traverseda 4y agoAnyone looking into this, I'd recommend python-playwright. Good browser engines and the ability to hot patch every request means you don't need to write a separate proxy to intercept requests. Writing something like this would be a pretty big project, at least a few solid weeks of work for someone who knew what they were doing, and there's a lot of edge cases.
- marginalia_nu 4y agoI run a crawler and I dunno if I agree with this description. Usually anti-bot measures are in place for a reason, either to protect some paywalled content from scraping, or to prevent bots from getting at forms and causing a mess. Few webmasters have cloudflare (or similar) anti-bot measures turned up higher than it needs to be (because it 100% will annoy people), and content that isn't a high-value target for blackhat bot activity is rarely difficult to crawl.