4 ms·
Exactly! While there are scraping APIs that unblock requests and charge for them, I'd love to learn more about how they work....
by r_singh 5y ago
Exactly!
While there are scraping APIs that unblock requests and charge for them, I'd love to learn more about how they work....
- deleted 5y ago[deleted]
- Eikon 5y agoThey use residential proxies with altered clients and / or headless browsers. Cloudflare's bot protection mostly makes use of TLS fingerprinting, and thus pretty easy to bypass.
- wswope 5y agoScraping is a cat and mouse game that’ll vary a lot by site. I’m far from an expert and welcome correction here, but the two big tricks that’ll go a long way AFAIK are using a residential proxy service (never tried one - they tend to be quite shady), and using a webdriver-type setup like Selenium or Puppeteer to mock realistic behavior (though IIRC you have to obfuscate both those systems since they’re detectable via JS).