3 ms·
This needs to be repeated because it helps to understand the issue. There are only four (4) search indexes of sufficient size in English, these are Bing, Googl
by ChuckMcM 8y ago
This needs to be repeated because it helps to understand the issue.
There are only four (4) search indexes of sufficient size in English, these are Bing, Google, Baidu (Chinese), and Yandex (Russian).
Everything on the internet that calls itself a "search engine" gets its search result data from one of those four indexes. Different front ends provide different features around the results, whether it is privacy enhancement, ad enhancement, Etc.
Google's English language index is generally considered the 'best' based on the depth of the index with respect to its crawl size and that index's ability to provide both good precision and recall.
You can use any search site you want but you will likely be using Google search results. If you feel like you're results aren't quite as good as Google's, and you are in the US, you are probably using Bing. Neither Baidu nor Yandex have US data centers so latency to their indexes is pretty bad for most front end web sites.
- m-p-3 8y agoDuckDuckGo have their own crawlers right?
- gbear605 8y agoThey used to, but now they don’t.
- arbie 8y agoDo you have a source link where I could read up on this? It would seem that DuckDuckBot is still alive and well.
- gbear605 8y agohttps://duck.co/help/results/ https://duck.co/help/results/ > We source from Bing, Yahoo, and Yandex.
- ChuckMcM 8y ago(which is kind of funny because Yahoo forwards (forwarded?) all their search traffic to Bing per their agreement with Microsoft when Microsoft didn't buy them. Not sure what Verizon does with that now.
- arbie 8y agoFrom https://duck.co/help/results/sources https://duck.co/help/results/sources: > DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we source from Bing, Yahoo, and Yandex. Interesting that DuckDuckBot is now relegated to Instant Answers.
- h2onock 8y agoThis is incorrect, not everything on the internet that calls itself a search engine get their results from one of those four. There's also Exalead, Gigablast, Mojeek and Findx for example. Disclaimer: I work for Mojeek so know that it only gets results from it's own index.
- ChuckMcM 8y agoThat is awesome love that you've crossed a billion pages in the index. I worked at Blekko (VP Eng/Ops) and at its peak it had about 5 billion pages indexed, and that is pretty good for targeted search. It loses on the long tail though for generalized search. Something we did at Blekko which was really interesting/helpful was to feed a search query to our index, Bing's, and Google's and then compare the results. That was a great way to compare notes on ranking strategies between the three engineering teams. [1] "In 2015, Mojeek's index surpassed the billion pages, the only search engine in the UK to accomplish this and one of just a handful worldwide to maintain an index of this size (English language). Although, passing this milestone is only the first step and Mojeek is continuing to increase its index size and improve relevancy." -- https://www.mojeek.com/about/technology.html https://www.mojeek.com/about/technology.html
- h2onock 8y agoThanks for the info. Marc (who created Mojeek) mentions Blekko a lot as he considers yourselves to be the last ones so far to have a proper go at creating a search engine from scratch. Also, we're now up to 2.2 billion pages and it's only a lack of servers that stops us from increasing at a much faster pace.
- ChuckMcM 8y agoFor what its worth, this is the ultimate calculus for a search engine; The revenue you can produce is proportional to your query rate, your query rate is proportional to your recurring user base, and your recurring user base is proportional to the usefulness of your searches, which is proportional to the number of pages in your index which is proportional to the number of servers you have running which is proportional to the cost of running the service. One of the things I would have done differently at Blekko is that I would have created our own in house ad serving system (network) way earlier in the life cycle (Blekko never did have an in house ad network). It gives you better control over advertising latency and the quality factor of the ads. You also get to keep all of the ad revenue rather than just some of it. And you can deal directly with advertisers rather than the ad-tech crowd which can be quite scummy at times. I wish you the best of luck in your endeavor, if you can crack the secret of surviving in a market where Google pays people over $4B/year just to send their search traffic to them, then you will be one of my heroes.