2 ms·
Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with you
by Aeolun 8y ago
Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.
- bogomipz 8y agoI'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?
- jdsully 8y agoA site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet. A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.