Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I twea

Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I tweak these values to what I estimate the website could handle (depending on how large it is). The page also outlines how throttling algorithm works, which you use as a starting point for your own work. https://docs.scrapy.org/en/latest/topics/autothrottle.html https://docs.scrapy.org/en/latest/topics/autothrottle.html