3 ms·
I have my crawlers running on a 24/7 notebook at my house because of that. No blocks whatever rate i use. Deployed to digital ocean runs for 5 minutes then bloc
by popotamonga 5y ago
I have my crawlers running on a 24/7 notebook at my house because of that. No blocks whatever rate i use. Deployed to digital ocean runs for 5 minutes then blocked forever.
- trutannus 5y agoEssentially. All you're doing here is making it easy for the target sites to mass block a bunch of scrapers. Running it off your device is generally better.