9 ms·

I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://new

by yegg 3y ago

I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807) but in short it isn't a binary thing and I think the fallacy people keep making is a modern search engine != web index. Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important. In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.

Waterluvian 3y ago
Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?
- lapcat 3y ago
  I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.
  - Waterluvian 3y ago
    That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
    - PeterisP 3y ago
      They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
      - Waterluvian 3y ago
        I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
        ribosometronome 3y ago
        I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
        Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
        PeterisP 3y ago
        You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.
deleted 3y ago
[deleted]
lxe 3y ago
Thanks for the update! From your other thread: > Bing is our largest source of traditional web links. I think "traditional web links" is the main (and I think should be the only) product of a search engine, and it seems like you rely on Bing quite significantly to serve search results.
- blowski 3y ago
  I use DDG as my main search engineer. Many of my searches have hashbangs into other sites. On DDG landing pages, I often go straight to the right hand side of the page, with the info panel. Sometimes, I click on traditional web links, but with less frequency as the sites are so often spam.
- yegg 3y ago
  From lots of experience, I can tell you if you don't have these things (and others, this is an incomplete list just off the top of my head) showing up in the right places (often above "traditional web links") at the right times then you cannot get long-term search engine adoption from mainstream users: - maps - directions - local business listings - about boxes - basic facts - conversions - news - videos - images - shopping - sports - stocks - weather - flights - entertainment (e.g. cast) - etc.
  - Darthy 3y ago
    Gabriel, could you please clarify once and for all: for "traditional web links", how much do you rely on bing? The user consensus seems to be 100%. What is the official number from you? Thanks!
    - ncallaway 3y ago
      The previous statement seems to answer that reasonably well. > We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways I don't know that it's reasonable to demand that they provide you an exact percentage.
kikokikokiko 3y ago
It's turtles all the way down, I mean, it's Bing. If your basic source of information, the thing that you rely on to create any smoke and mirrors on top of it, i.e. the index, is basically Bing and just Bing, then you're just a Bing mirror. More and more I see myself using Yandex out of all things to get decent results. The extremes the web got in 2023 are bizarre, the "best" search engine in the world today may very well come from a dictatorship controlled country.
- photonerd 3y ago
  Either you’re very misinformed about what a mirror is, or you’re being extraordinarily disingenuous here. Bing is their initial and primary source of data for web links. Not the only one either, but the primary one. They maintain their own index from it. That’s not “a mirror” by any stretch of the imagination. What appears to have happened here is some form of replication bug where in a way yet to be determined a removal from the original data source was unintentionally replicated. Data synchronization is hard when you’re not directly mirroring data. This kind of bug crops up
  - kikokikokiko 3y ago
    If they're not just a mirror, it seems rather strange that when a specific website disappears from Bing's index, it also disappears from DDG. In my very common sense based view of the situation, if DDG really builds it's own "index" based on a variety of sources, then why did TechDirt completely disappeared from it's search results? DDG can argue all day about how much their "local and knowledge graph" is built in-house, yada yada yada. At the end of the day, a search engine must retrieve the most relevant websites for a given query. And nothing can be more relevant to "site:xxx.xxx.xxx" than the site itself. If it doesn't appear on DDG results, then it's not indexed. And then it clearly shows that DDG is nothing but a fancy mirror to Bing search.
    - photonerd 3y ago
      > it seems rather strange […] Not really. That’s the point. Data replication is hard, especially when you have to consistently update from the same data source. Bugs happen.
WheatMillington 3y ago
You can spin it any way you like, but the fact is evident: you rely on Bing to the extent that if Bing delists a site, that site will be completely sanitised from every single part of a DDG search, including all those dozens of modules you keep referring to.

by yegg 3y ago

Waterluvian 3y ago

Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?

lapcat 3y ago
I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.
- Waterluvian 3y ago
  That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
  - PeterisP 3y ago
    They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
    - Waterluvian 3y ago
      I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
      - ribosometronome 3y ago
        I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
        Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
        PeterisP 3y ago
        You can diff indices but it doesn't make economic sense to do so. The whole point of using Bing as the base is so that you don't have to make and keep an index of everything yourself, so DDG wouldn't have another exhaustive index to compare to (and if they would, they wouldn't need Bing). They index some things to add onto Bing, but they don't try to capture everything. Furthermore, any specific site missing from the index does not necessarily mean a flaw - okay, you would diff some indices, and find out a long list of what Bing has excluded... and then what? For every 'fixable censorship' case (accidental or intentional) there will be thousands of spam or malicious sites which should be excluded, as nowadays the key part of search is not finding everything but throwing out the results which want to be found but shouldn't. Again, DDG wants to piggyback on the effort that Bing is doing to filter the index, and if they want to second-guess all Bing's filtering (as opposed to just making a fix for this specific case) then they have to replicate and improve on all of the (huge and expensive!) Bing's filtering effort; which goes counter to the reason for using Bing which is to avoid all this expense.

deleted 3y ago

[deleted]

lxe 3y ago

Thanks for the update! From your other thread: > Bing is our largest source of traditional web links. I think "traditional web links" is the main (and I think should be the only) product of a search engine, and it seems like you rely on Bing quite significantly to serve search results.

blowski 3y ago
I use DDG as my main search engineer. Many of my searches have hashbangs into other sites. On DDG landing pages, I often go straight to the right hand side of the page, with the info panel. Sometimes, I click on traditional web links, but with less frequency as the sites are so often spam.
yegg 3y ago
From lots of experience, I can tell you if you don't have these things (and others, this is an incomplete list just off the top of my head) showing up in the right places (often above "traditional web links") at the right times then you cannot get long-term search engine adoption from mainstream users: - maps - directions - local business listings - about boxes - basic facts - conversions - news - videos - images - shopping - sports - stocks - weather - flights - entertainment (e.g. cast) - etc.
- Darthy 3y ago
  Gabriel, could you please clarify once and for all: for "traditional web links", how much do you rely on bing? The user consensus seems to be 100%. What is the official number from you? Thanks!
  - ncallaway 3y ago
    The previous statement seems to answer that reasonably well. > We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways I don't know that it's reasonable to demand that they provide you an exact percentage.

kikokikokiko 3y ago

It's turtles all the way down, I mean, it's Bing. If your basic source of information, the thing that you rely on to create any smoke and mirrors on top of it, i.e. the index, is basically Bing and just Bing, then you're just a Bing mirror. More and more I see myself using Yandex out of all things to get decent results. The extremes the web got in 2023 are bizarre, the "best" search engine in the world today may very well come from a dictatorship controlled country.

photonerd 3y ago
Either you’re very misinformed about what a mirror is, or you’re being extraordinarily disingenuous here. Bing is their initial and primary source of data for web links. Not the only one either, but the primary one. They maintain their own index from it. That’s not “a mirror” by any stretch of the imagination. What appears to have happened here is some form of replication bug where in a way yet to be determined a removal from the original data source was unintentionally replicated. Data synchronization is hard when you’re not directly mirroring data. This kind of bug crops up
- kikokikokiko 3y ago
  If they're not just a mirror, it seems rather strange that when a specific website disappears from Bing's index, it also disappears from DDG. In my very common sense based view of the situation, if DDG really builds it's own "index" based on a variety of sources, then why did TechDirt completely disappeared from it's search results? DDG can argue all day about how much their "local and knowledge graph" is built in-house, yada yada yada. At the end of the day, a search engine must retrieve the most relevant websites for a given query. And nothing can be more relevant to "site:xxx.xxx.xxx" than the site itself. If it doesn't appear on DDG results, then it's not indexed. And then it clearly shows that DDG is nothing but a fancy mirror to Bing search.
  - photonerd 3y ago
    > it seems rather strange […] Not really. That’s the point. Data replication is hard, especially when you have to consistently update from the same data source. Bugs happen.

WheatMillington 3y ago

You can spin it any way you like, but the fact is evident: you rely on Bing to the extent that if Bing delists a site, that site will be completely sanitised from every single part of a DDG search, including all those dozens of modules you keep referring to.