3 ms·
RE: pooling - yup, pooling for the win! As a self taught programmer, this was one block of code I was particularly proud of :)
by chptung 4y ago
RE: pooling - yup, pooling for the win! As a self taught programmer, this was one block of code I was particularly proud of :)
- zasdffaa 4y agoCrawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and the sensitivity (you mention saleable stuff in another comment so this might well make a difference). FYI. I did a reasonable amount of scraping myself. (err, crawling = scraping? or maybe they different things) Edit: tidy + clarify