2 ms·
Google also plays nice. I manage a few sites that get hammered by scrapers to the extent that it causes big spikes in CPU, something that doesn't happen with G.
by stef25 5y ago
Google also plays nice. I manage a few sites that get hammered by scrapers to the extent that it causes big spikes in CPU, something that doesn't happen with G.
- that_guy_iain 5y agoSo many people don't even think about number of requests and often want the job to end as quickly as possible. For me, that is thinking short term, I've seen requests per second as low as 4 cause performance issues for a company. If you want to scrape long term you need to think requests per minute. That is unless you're dealing with a mega site then it's keeping a low request per second.