Across many requests, Google has a very prevalent "bandwidth/time" calculation for each domain/subdomain that factors in domain importance, # of pages, frequenc

Across many requests, Google has a very prevalent "bandwidth/time" calculation for each domain/subdomain that factors in domain importance, # of pages, frequency of page updates, and even input from webmasters via Webmaster Tools. The bot will just stop requesting pages after that ratio is reached. (This is why new domains that publish LOTS of pages at once may take a while to get crawled.) Across individual requests, they have distanced themselves from setting a concrete limit to request sizes[1] (they used to only cache the first 100Kb, then people saw them caching up to 400Kb, now they definitely index things like PDFs that are much larger.) [1] http://www.mattcutts.com/blog/how-many-links-per-page/ http://www.mattcutts.com/blog/how-many-links-per-page/