3 ms·

I'd agree. As somebody scraping content, what's so bad about increasing the timeout to like 10 seconds? That way the servers can handle the traffic easily and y

I'd agree. As somebody scraping content, what's so bad about increasing the timeout to like 10 seconds? That way the servers can handle the traffic easily and you're not being a jerk. If you have one async thread for each domain you can still get lots of data quickly. Causing a denial of service attack is very avoidable.

jacquesm 8y ago
And you're going to be hitting millions of hosts anyway, so all you have to do is rotate from one host to the next and randomize your worker queues. It might take a little longer but you will not blow up someone's aging server. Being a good citizen of the net means to take into account that even if you have gigabits of bandwidth to burn the counterparty may not (and could easily be on the sharp end of a bandwidth capped contract).