4 ms·
> Don't be too kind on the big websites. I usually recommend latency-based dynamic load control for that. Once the website starts to reply 500-1000ms longer th
by isoos 7y ago
> Don't be too kind on the big websites.
I usually recommend latency-based dynamic load control for that. Once the website starts to reply 500-1000ms longer than the average one-thread latency, it is time to take a bit of it back. It is also a co-operative strategy between fellow scrapers, even if they don't know about the other ones pushing larger load on the servers.
- arcticfox 7y ago1000ms is a massive slowdown when revenue-noticeable impacts are far, far smaller. I don't know the legality, but hitting a site hard enough to cause 1000ms slowdowns seems like it's approaching DOS legality issues.