4 ms·

I wonder how Google handles traffic to its crawlers. A webpage I viewed recently loaded an entire database query into my web browser, then I made queries locall

I wonder how Google handles traffic to its crawlers. A webpage I viewed recently loaded an entire database query into my web browser, then I made queries locally to sort it, which kinda sucked since 20MB of information is a lot. I figure Google has to have some form of safeguard against this. Either CPU or network bandwidth limited, likely a time limit too. In my head I'm picturing a crawler locked in a loop of forever querying random google search results and adding them to a page.

andrenotgiant 12y ago
Across many requests, Google has a very prevalent "bandwidth/time" calculation for each domain/subdomain that factors in domain importance, # of pages, frequency of page updates, and even input from webmasters via Webmaster Tools. The bot will just stop requesting pages after that ratio is reached. (This is why new domains that publish LOTS of pages at once may take a while to get crawled.) Across individual requests, they have distanced themselves from setting a concrete limit to request sizes[1] (they used to only cache the first 100Kb, then people saw them caching up to 400Kb, now they definitely index things like PDFs that are much larger.) [1] http://www.mattcutts.com/blog/how-many-links-per-page/ http://www.mattcutts.com/blog/how-many-links-per-page/