4 ms·
Yeah, no offence, but sleep(2 + random.sample(coffee + toilet + sneezing + normal response time)) has been a required part of web scrapers since forever. With
by candiodari 3y ago
Yeah, no offence, but sleep(2 + random.sample(coffee + toilet + sneezing + normal response time)) has been a required part of web scrapers since forever.
With coffee N(1,5 minutes, 20 seconds), toilet N(4 minutes, 30 seconds), ...
- rezonant 3y agoI guess it depends on how you're scraping. For general web crawling, simply implementing a response time based crawl back off per origin and identifying yourself appropriately in User Agent goes a long way. If you are instead automating Facebook's SPA to pull comments for analysis, then yeah you need to emulate a human, because that's not how they intend you to do it.
- jacquesm 3y agoThat's incredibly clever!