4 ms·
A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).
by mark_l_watson 12y ago
A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).
- thomasfoster96 12y agoCuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index. What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.
- boyter 12y agoPerhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though. The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.
- bobajeff 12y agoI think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
- boyter 12y agoAlso true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top. I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches. I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.
- metronius 12y agoIf you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
- psykovsky 12y agoYou say that like it doesn't happen all the time. googlebot included.
- sytelus 12y agoCommon Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For anything under ~10B pages is most likely to be called "research" or "toy" index. You have almost zero chance of building competitive index if you are below ~15B. Google is rumored to crawl about 150B pages as reference. There are estimated 1.2T pages out there which does not get crawled due to various reasons.
- metronius 12y agoI dont think that you need so much pages - most of 150B pages will be never ever shown displayed at SERP.
- Arnt 12y agoTrue, you just need a subset. Now how you you identify that subset without indexing the pages to find out whether each page is in the subset you need? IIRC google used to scan different pages at very different frequencies. Quite possibly because it has assigns pages into subsets every time it indexes.