> Google's bot was one of the few well behaved ones and would even slow scraping if it saw a spike in the response times. Google has invested decades of core r

> Google's bot was one of the few well behaved ones and would even slow scraping if it saw a spike in the response times. Google has invested decades of core research with an army of PhDs into its crawler, particularly around figuring out when to recrawl a page. For example (a bit dated, but you can follow the refs if you're interested): https://www.niss.org/sites/default/files/Tassone_interface6.pdf https://www.niss.org/sites/default/files/Tassone_interface6....