At Google's scale any performance drop can have massive implications. If Google's crawl rate is 100 million[1] pages a day then a 1% drop in crawl rate means 1

At Google's scale any performance drop can have massive implications. If Google's crawl rate is 100 million[1] pages a day then a 1% drop in crawl rate means 1 million less pages crawled per day (which has many implications, for example having to use more compute power to regain crawl rate which raises costs etc.) You are right that it is not a real-time exercise but they do have crawl targets. You cannot be flippant about "Why don't they just do X" when scale is that big. [1] Picked out of the air but probably in the right magnitude (or even a little small)