4 ms·
On this note, does anyone know of a drip/gradual scrape process for not overloading a host's servers, such that the process could run over the span of a day/wee
by tenkabuto 9y ago
On this note, does anyone know of a drip/gradual scrape process for not overloading a host's servers, such that the process could run over the span of a day/week? I'm not worried about the content changing rapidly.
- gwbas1c 9y agoUhhh, have a brain and common sense when you write your scraper? The simplest thing to do is NOT parallelize your scraper, and sleep for a few seconds between requests.