3 ms·

Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's be

Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?

lapcat 3y ago
I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.
- Waterluvian 3y ago
  That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
  - PeterisP 3y ago
    They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
    - Waterluvian 3y ago
      I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
      - ribosometronome 3y ago
        I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
        Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
        PeterisP 3y ago
        You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.

Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?

lapcat 3y ago

I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.

Waterluvian 3y ago
That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
- PeterisP 3y ago
  They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
  - Waterluvian 3y ago
    I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
    - ribosometronome 3y ago
      I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
      - Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
        PeterisP 3y ago
        You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.