No one brought it up so I will, Google has an algorithm which measures responsiveness of a site to adjust crawl speed. Google will adjust the crawl rate so it w

No one brought it up so I will, Google has an algorithm which measures responsiveness of a site to adjust crawl speed. Google will adjust the crawl rate so it won't negatively impact performance. It will also crawl your site's pages more based on updates to content and popularity of URLs. I've written crawlers, it's not as trivial as some would like to believe, but analyzing the content to provide relevant search results is 99.99% of the effort of the Google/Bing teams.