3 ms·
Not following how this is helpful? From what I've seen the cloud IPs are the first to get blocked by anti-scraping tech. e.g. Built an amazon scraping thing a
by Havoc 5y ago
Not following how this is helpful?
From what I've seen the cloud IPs are the first to get blocked by anti-scraping tech.
e.g. Built an amazon scraping thing at home. Worked fine. Deployed to cloud. Literally first request Amazon goes nope you're a bot
- popotamonga 5y agoI have my crawlers running on a 24/7 notebook at my house because of that. No blocks whatever rate i use. Deployed to digital ocean runs for 5 minutes then blocked forever.
- trutannus 5y agoEssentially. All you're doing here is making it easy for the target sites to mass block a bunch of scrapers. Running it off your device is generally better.