3 ms·
They crawl the web non-stop yes, but how often an individual site is crawled varies. > Our crawl process is algorithmic: computer programs determine how often
by codetrotter 8y ago
They crawl the web non-stop yes, but how often an individual site is crawled varies.
> Our crawl process is algorithmic: computer programs determine how often we should crawl each site. If you post new articles on your site throughout the day, you should be able to see them appear on Google News fairly quickly.
> [...]
> From the moment we discover a new article, we'll keep re-crawling it looking for changes. Since we noticed that most changes to articles occur just after they're published, we revisit articles most frequently in the first day after we've found them. After that, we visit them less often.
https://support.google.com/news/publisher-center/answer/40392?hl=en https://support.google.com/news/publisher-center/answer/4039...
- askmike 8y agoFair point, if your content gets copied to unknown website X which Google hasn't crawled for days they might not be able to tell where the article appeared first.