4 ms·
The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an avail
by jedberg 9y ago
The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an available slot, it crawls the next site (obviously that's simplified).
Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site.
If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.
- owenversteeg 9y agoThat's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
- himlion 9y agoI think that would lend itself to abuse.