4 ms·
Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though. Th
by boyter 12y ago
Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though.
The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.
- bobajeff 12y agoI think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
- boyter 12y agoAlso true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top. I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches. I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.
- metronius 12y agoIf you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
- psykovsky 12y agoYou say that like it doesn't happen all the time. googlebot included.