4 ms·

I know Cuil is the poster-boy of how to suck at launch but it really is a pity they didn't succeed. With Bing powering Yahoo there are really only 4 players in

by boyter 15y ago

I know Cuil is the poster-boy of how to suck at launch but it really is a pity they didn't succeed. With Bing powering Yahoo there are really only 4 players in the English language index space, Google, Bing, Gigablast and Blekko. Add in the non English ones Yandex and Baidu and there really is a lack of companies to compete with. Its because of this I want DuckDuckGo or another company to eventually do its own deep indexing to avoid the very singular view of the web we are presented with.

tripzilch 15y ago
Yandex is pretty good for generic search btw, even if you're not in Russia. They even got image search. Also, a HN-familiar orange colour scheme ;-)
- boyter 15y ago
  Had a play then and the results are quite good. Using a sample search for viagra on DDG, Google, Bing, Yandex, Gigablast and Blekko it seems to have less spam in comparison to all but Blekko.
  - dmoy 15y ago
    That is an interesting query to test for spam. Now that I think about it though, I could see how that could be a start for a pretty excellent benchmark for filtering, given the ratio of viagra-to-not-viagra spam in my spam box.
    - boyter 15y ago
      Its always been my goto for working out if a search engine has any credibility or not. You do get odd looks when you have 5+ windows open all searching for viagra in the office though!
riffraff 15y ago
isn't Ask still working and much bigger than blekko or gigablast?
- boyter 15y ago
  Ah good point I had forgotten ask. Do they still maintain their own index? I recall a while ago they were discussing dropping it as a cost saver. EDIT - Seems they no longer do. http://searchengineland.com/ask-com-to-focus-on-qa-search-end-web-crawling-55209 http://searchengineland.com/ask-com-to-focus-on-qa-search-en... Apparently they have been using someone else's index for some time, but maintain their own over specific question/answer sites.
bobds 15y ago
http://commoncrawl.org/ http://commoncrawl.org/ Common Crawl is a non-profit that maintains a big crawl dataset.
- boyter 15y ago
  That's only the first portion of the problem, and in my opinion not the most difficult. Anyone can write a web crawler since its essentially, while(links) { get(link) } With a little xargs or parallel magic + wget you can suck down billions of pages with little effort. There is more to it then that to keep things fresh but its not that hard to just kick it off now and then. The thing that's difficult is taking that data and turning it into an index able to span multiple machines. If it was that easy DuckDuckGo or someone else would have taken that data and done something with it by now.

by boyter 15y ago

tripzilch 15y ago

Yandex is pretty good for generic search btw, even if you're not in Russia. They even got image search. Also, a HN-familiar orange colour scheme ;-)

boyter 15y ago
Had a play then and the results are quite good. Using a sample search for viagra on DDG, Google, Bing, Yandex, Gigablast and Blekko it seems to have less spam in comparison to all but Blekko.
- dmoy 15y ago
  That is an interesting query to test for spam. Now that I think about it though, I could see how that could be a start for a pretty excellent benchmark for filtering, given the ratio of viagra-to-not-viagra spam in my spam box.
  - boyter 15y ago
    Its always been my goto for working out if a search engine has any credibility or not. You do get odd looks when you have 5+ windows open all searching for viagra in the office though!

riffraff 15y ago

isn't Ask still working and much bigger than blekko or gigablast?

boyter 15y ago
Ah good point I had forgotten ask. Do they still maintain their own index? I recall a while ago they were discussing dropping it as a cost saver. EDIT - Seems they no longer do. http://searchengineland.com/ask-com-to-focus-on-qa-search-end-web-crawling-55209 http://searchengineland.com/ask-com-to-focus-on-qa-search-en... Apparently they have been using someone else's index for some time, but maintain their own over specific question/answer sites.

bobds 15y ago

http://commoncrawl.org/ http://commoncrawl.org/ Common Crawl is a non-profit that maintains a big crawl dataset.

boyter 15y ago
That's only the first portion of the problem, and in my opinion not the most difficult. Anyone can write a web crawler since its essentially, while(links) { get(link) } With a little xargs or parallel magic + wget you can suck down billions of pages with little effort. There is more to it then that to keep things fresh but its not that hard to just kick it off now and then. The thing that's difficult is taking that data and turning it into an index able to span multiple machines. If it was that easy DuckDuckGo or someone else would have taken that data and done something with it by now.