3 ms·
CDNs like Distil Networks and Cloudflare make scraping more difficult than it used to be. If you get caught by them, you can end up blocked from all of the site
by twblalock 9y ago
CDNs like Distil Networks and Cloudflare make scraping more difficult than it used to be. If you get caught by them, you can end up blocked from all of the sites they protect, not just the one you were scraping.
- always_good 9y agoWriting some scrapers this week, I noticed it's also common for the origin server to just check if the request is coming from VPN/VPS IP address range. For example, the exact same request will work from your home connection where it doesn't work from EC2.