3 ms·
Google crawls the open web non stop. They definitely have this data. It's just not exposed to the (legal) team dealing with DMCA takedown requests.. EDIT: Or a
by askmike 8y ago
Google crawls the open web non stop. They definitely have this data. It's just not exposed to the (legal) team dealing with DMCA takedown requests..
EDIT: Or as to the example above: whatever application is responsible for picking the main article to show somewhere.
- codetrotter 8y agoThey crawl the web non-stop yes, but how often an individual site is crawled varies. > Our crawl process is algorithmic: computer programs determine how often we should crawl each site. If you post new articles on your site throughout the day, you should be able to see them appear on Google News fairly quickly. > [...] > From the moment we discover a new article, we'll keep re-crawling it looking for changes. Since we noticed that most changes to articles occur just after they're published, we revisit articles most frequently in the first day after we've found them. After that, we visit them less often. https://support.google.com/news/publisher-center/answer/40392?hl=en https://support.google.com/news/publisher-center/answer/4039...
- askmike 8y agoFair point, if your content gets copied to unknown website X which Google hasn't crawled for days they might not be able to tell where the article appeared first.