4 ms·
I would argue that the proper implementation provides real rate limiting, both in terms of max requests per second and also max concurrent requests. Limiting to
by zevyoura 14y ago
I would argue that the proper implementation provides real rate limiting, both in terms of max requests per second and also max concurrent requests. Limiting to one concurrent request is likely to be extremely slow for any significant amount of data, and a couple concurrent requests is not impolite. Obviously I'm not saying you should effectively DoS the site you're scraping, but there's a balance and 1 concurrent request is almost definitely the wrong place to set it.