4 ms·
Common Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For
by sytelus 12y ago
Common Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For anything under ~10B pages is most likely to be called "research" or "toy" index. You have almost zero chance of building competitive index if you are below ~15B. Google is rumored to crawl about 150B pages as reference. There are estimated 1.2T pages out there which does not get crawled due to various reasons.
- metronius 12y agoI dont think that you need so much pages - most of 150B pages will be never ever shown displayed at SERP.
- Arnt 12y agoTrue, you just need a subset. Now how you you identify that subset without indexing the pages to find out whether each page is in the subset you need? IIRC google used to scan different pages at very different frequencies. Quite possibly because it has assigns pages into subsets every time it indexes.