4 ms·
That wouldn't make any sense given that DDG is thin UX veneer around Bing, and it doesn't maintain its own index.
by 0xbear 9y ago
That wouldn't make any sense given that DDG is thin UX veneer around Bing, and it doesn't maintain its own index.
- dmead 9y agois that true? i thought DDG predates bing?
- 0xbear 9y agoI believe they initially used Google as their backend. Don't quote me on that though. One thing I know for sure is their search is not really provided by them.
- SyneRyder 9y agoApparently DuckDuckGo now gets its web site results from a combination of Bing, Yahoo and Yandex. Its focus now seems to be on aggregating results from hundreds of niche search engines and choosing the best engine for a particular search. https://duck.co/help/results/sources https://duck.co/help/results/sources
- 0xbear 9y agoAnd Yahoo, in turn, gets its results from Bing.
- ChuckMcM 9y agoExactly, there are two commonly used english indexes, Bing and Google. Yandex has one but they closed their local data center so it has a lot of lag getting to it from the US. Tsignal [1] claims to be building their own index, which if they are would mean there was a third index to choose from. [1] https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/ https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/
- 0xbear 9y agoPeople seem surprised, but it's perfectly predictable. To maintain an index of sufficient quality and size costs billions of dollars a year. Heck, engineering cost alone is probably $500M for MS, and more for Google. DGG doesn't even come close to having the table stakes to play this game. Another area where table stakes are incredibly high is geo (maps).
- ChuckMcM 9y agoActually, at Blekko it cost us about $2M/year to hold an index with 5 billion documents, it would have cost use less than $10M/year to hold an index of 10 billion documents (this is suggested by many to be the 'sweet spot' of coverage) to have that many documents you will create a crawl frontier of between 500B and 1T URIs.
- 0xbear 9y ago“Holding” an index is the trivial part of a search engine. That’s something a single Google engineer can put together in a week (no crawler, just storing and primitive ranker). What’s going to kill you is ranking (especially in the tail) and keeping the index fresh, including the tail. Ranking requires human input. Keeping it fresh requires some non-trivial effort in the crawler too. And then there’s also serving all of it at 1M QPS, as a prerequisite for creating enough ad revenue to sustain the effort. That requires predictive caching. As a result you get a couple thousand people working on the thing, each pulling down about $400k on average when you total everything up. That’s $800M right off the bat, unless you cut corners like Bing (cleverly) did by stealing your competitors clicks as weak relevance supervision.
- losvedir 9y agoLast time I asked that question, someone posted this interesting comparison screenshot of their search on Bing and DDG: https://imgur.com/P1akOab https://imgur.com/P1akOab