3 ms·

I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 tr

I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).

Waterluvian 3y ago
I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
- PeterisP 3y ago
  You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.

Waterluvian 3y ago

I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?

PeterisP 3y ago
You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.