3 ms·
My impression, based on eyeballing logs for my own low traffic, unimportant sites: Google collects and uses stats on how often pages change, how often they come
by dwc 14y ago
My impression, based on eyeballing logs for my own low traffic, unimportant sites: Google collects and uses stats on how often pages change, how often they come up in searches, etc., and for static and/or low importance pages they scale crawling back. This frees up crawling resources to spend elsewhere, like on important, frequently updated pages, new sites, et al. I haven't looked closely in a couple of years, but the pattern used to be that new content would be crawled fairly agressively and after a while it would back off to an "appropriate" level.
Back then, other search engines took a good while to begin crawling my new stuff in earnest, and once they did their crawling pattern did not seem to change much over time between a (relatively) popular changing page vs. a lonely static page.