5 ms·

And Yahoo, in turn, gets its results from Bing.

by 0xbear 9y ago

ChuckMcM 9y ago
Exactly, there are two commonly used english indexes, Bing and Google. Yandex has one but they closed their local data center so it has a lot of lag getting to it from the US. Tsignal [1] claims to be building their own index, which if they are would mean there was a third index to choose from. [1] https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/ https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/
- 0xbear 9y ago
  People seem surprised, but it's perfectly predictable. To maintain an index of sufficient quality and size costs billions of dollars a year. Heck, engineering cost alone is probably $500M for MS, and more for Google. DGG doesn't even come close to having the table stakes to play this game. Another area where table stakes are incredibly high is geo (maps).
  - ChuckMcM 9y ago
    Actually, at Blekko it cost us about $2M/year to hold an index with 5 billion documents, it would have cost use less than $10M/year to hold an index of 10 billion documents (this is suggested by many to be the 'sweet spot' of coverage) to have that many documents you will create a crawl frontier of between 500B and 1T URIs.
    - 0xbear 9y ago
      “Holding” an index is the trivial part of a search engine. That’s something a single Google engineer can put together in a week (no crawler, just storing and primitive ranker). What’s going to kill you is ranking (especially in the tail) and keeping the index fresh, including the tail. Ranking requires human input. Keeping it fresh requires some non-trivial effort in the crawler too. And then there’s also serving all of it at 1M QPS, as a prerequisite for creating enough ad revenue to sustain the effort. That requires predictive caching. As a result you get a couple thousand people working on the thing, each pulling down about $400k on average when you total everything up. That’s $800M right off the bat, unless you cut corners like Bing (cleverly) did by stealing your competitors clicks as weak relevance supervision.
      - ChuckMcM 9y ago
        Sorry if I wasn't clear, at Blekko we had an index at our peak of 5 billion documents. And we had the full stack to crawl, rank, cleanse, and search that index as well. The crawl frontier was about 300 billion URIs. That was what I meant by "holding".
  - earenndil 9y ago
    Findx[1] has its own index and afaik they don't have very much money. DDG obviously has more users, so they have to deal with the number of requests, but that sort of thing scales pretty well. 1: https://www.findx.com/ https://www.findx.com/

And Yahoo, in turn, gets its results from Bing.

by 0xbear 9y ago

And Yahoo, in turn, gets its results from Bing.

Exactly, there are two commonly used english indexes, Bing and Google. Yandex has one but they closed their local data center so it has a lot of lag getting to it from the US. Tsignal [1] claims to be building their own index, which if they are would mean there was a third index to choose from. [1] https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/ https://www.theregister.co.uk/2017/08/31/deepsearch_tsignal/

0xbear 9y ago
People seem surprised, but it's perfectly predictable. To maintain an index of sufficient quality and size costs billions of dollars a year. Heck, engineering cost alone is probably $500M for MS, and more for Google. DGG doesn't even come close to having the table stakes to play this game. Another area where table stakes are incredibly high is geo (maps).
- ChuckMcM 9y ago
  Actually, at Blekko it cost us about $2M/year to hold an index with 5 billion documents, it would have cost use less than $10M/year to hold an index of 10 billion documents (this is suggested by many to be the 'sweet spot' of coverage) to have that many documents you will create a crawl frontier of between 500B and 1T URIs.
  - 0xbear 9y ago
    “Holding” an index is the trivial part of a search engine. That’s something a single Google engineer can put together in a week (no crawler, just storing and primitive ranker). What’s going to kill you is ranking (especially in the tail) and keeping the index fresh, including the tail. Ranking requires human input. Keeping it fresh requires some non-trivial effort in the crawler too. And then there’s also serving all of it at 1M QPS, as a prerequisite for creating enough ad revenue to sustain the effort. That requires predictive caching. As a result you get a couple thousand people working on the thing, each pulling down about $400k on average when you total everything up. That’s $800M right off the bat, unless you cut corners like Bing (cleverly) did by stealing your competitors clicks as weak relevance supervision.
    - ChuckMcM 9y ago
      Sorry if I wasn't clear, at Blekko we had an index at our peak of 5 billion documents. And we had the full stack to crawl, rank, cleanse, and search that index as well. The crawl frontier was about 300 billion URIs. That was what I meant by "holding".
- earenndil 9y ago
  Findx[1] has its own index and afaik they don't have very much money. DDG obviously has more users, so they have to deal with the number of requests, but that sort of thing scales pretty well. 1: https://www.findx.com/ https://www.findx.com/