4 ms·

I thought you barely relied on Bing anymore. At least, that's the claim you've repeatedly made here. This indicates that DDG is little more than a Bing mirror.

kats 3y ago
[flagged]
yegg 3y ago
I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807) but in short it isn't a binary thing and I think the fallacy people keep making is a modern search engine != web index. Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important. In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.
- Waterluvian 3y ago
  Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?
  - lapcat 3y ago
    I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.
    - Waterluvian 3y ago
      That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
      - PeterisP 3y ago
        They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
        Waterluvian 3y ago
        I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
        ribosometronome 3y ago
        I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
        Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
- deleted 3y ago
  [deleted]
- lxe 3y ago
  Thanks for the update! From your other thread: > Bing is our largest source of traditional web links. I think "traditional web links" is the main (and I think should be the only) product of a search engine, and it seems like you rely on Bing quite significantly to serve search results.
  - blowski 3y ago
    I use DDG as my main search engineer. Many of my searches have hashbangs into other sites. On DDG landing pages, I often go straight to the right hand side of the page, with the info panel. Sometimes, I click on traditional web links, but with less frequency as the sites are so often spam.
  - yegg 3y ago
    From lots of experience, I can tell you if you don't have these things (and others, this is an incomplete list just off the top of my head) showing up in the right places (often above "traditional web links") at the right times then you cannot get long-term search engine adoption from mainstream users: - maps - directions - local business listings - about boxes - basic facts - conversions - news - videos - images - shopping - sports - stocks - weather - flights - entertainment (e.g. cast) - etc.
    - Darthy 3y ago
      Gabriel, could you please clarify once and for all: for "traditional web links", how much do you rely on bing? The user consensus seems to be 100%. What is the official number from you? Thanks!
      - ncallaway 3y ago
        The previous statement seems to answer that reasonably well. > We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways I don't know that it's reasonable to demand that they provide you an exact percentage.
- kikokikokiko 3y ago
  It's turtles all the way down, I mean, it's Bing. If your basic source of information, the thing that you rely on to create any smoke and mirrors on top of it, i.e. the index, is basically Bing and just Bing, then you're just a Bing mirror. More and more I see myself using Yandex out of all things to get decent results. The extremes the web got in 2023 are bizarre, the "best" search engine in the world today may very well come from a dictatorship controlled country.
  - photonerd 3y ago
    Either you’re very misinformed about what a mirror is, or you’re being extraordinarily disingenuous here. Bing is their initial and primary source of data for web links. Not the only one either, but the primary one. They maintain their own index from it. That’s not “a mirror” by any stretch of the imagination. What appears to have happened here is some form of replication bug where in a way yet to be determined a removal from the original data source was unintentionally replicated. Data synchronization is hard when you’re not directly mirroring data. This kind of bug crops up
    - kikokikokiko 3y ago
      If they're not just a mirror, it seems rather strange that when a specific website disappears from Bing's index, it also disappears from DDG. In my very common sense based view of the situation, if DDG really builds it's own "index" based on a variety of sources, then why did TechDirt completely disappeared from it's search results? DDG can argue all day about how much their "local and knowledge graph" is built in-house, yada yada yada. At the end of the day, a search engine must retrieve the most relevant websites for a given query. And nothing can be more relevant to "site:xxx.xxx.xxx" than the site itself. If it doesn't appear on DDG results, then it's not indexed. And then it clearly shows that DDG is nothing but a fancy mirror to Bing search.
      - photonerd 3y ago
        > it seems rather strange […] Not really. That’s the point. Data replication is hard, especially when you have to consistently update from the same data source. Bugs happen.
- WheatMillington 3y ago
  You can spin it any way you like, but the fact is evident: you rely on Bing to the extent that if Bing delists a site, that site will be completely sanitised from every single part of a DDG search, including all those dozens of modules you keep referring to.
FormerBandmate 3y ago
Why is Bing doing this? It feels like we’re burying the lede, a $2 trillion company isn’t showing sites that are critical of it on its supposedly unbiased search engine
- criddell 3y ago
  Does Microsoft actually claim Bing is unbiased?
pb7 3y ago
Regardless of what their CEO claims here every single time, if Bing cut off their access, DDG would be dead overnight. All the stuff they claim to do themselves is <1% of whatever value DDG provides. Lucky for them it’s just enough to be able to use terms like “largely” instead of “entirely”, “other sources” instead of “only source”, etc.
- photonerd 3y ago
  They wouldn’t die overnight at all. They would fast become outdated however, but that’s a different problem. One solvable with another source.

kats 3y ago

[flagged]

yegg 3y ago

I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807) but in short it isn't a binary thing and I think the fallacy people keep making is a modern search engine != web index. Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important. In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.

Waterluvian 3y ago
Thanks for looking into this. Would love to learn how a specific query like "site: techdirt.com" got wiped like that. Will you share your findings once it's been identified and resolved?
- lapcat 3y ago
  I doubt there will be any findings, because the removal from the index was done by Microsoft rather than DuckDuckGo. The postmortem has to come from Microsoft. Likely the only thing that Yegg can do is poke some internal contact at MS and tell them to fix it.
  - Waterluvian 3y ago
    That's a finding, though. "We still depend too much on Bing and so when you search for a website that Microsoft has censored, we can't do any better."
    - PeterisP 3y ago
      They can do better, however, it takes time and effort to do that - they can and do augment Bing results with other data sources when needed, but up until recently there was no need to add anything extra with respect to Techdirt; it certainly makes no sense to index all of the internet and try to duplicate all the search results for every search just in case Bing has filtered something out.
      - Waterluvian 3y ago
        I feel like that's exactly why they should duplicate all the effort. So that they aren't thrall to Microsoft's choices.
        ribosometronome 3y ago
        I feel like they would love to be able to do that. One potentially relevant line from the article: >I love that first one. Microsoft, a company with a $2.5 trillion market cap, “may not have enough resources” to crawl and index Techdirt? Cool. I have to imagine DDG's valuation wouldn't quite hit 2.5 trillion if it went public. It's not absurd that they aren't able to fully duplicate those efforts (or that they don't need to in order to have a decent product).
        Waterluvian 3y ago
        I mean… I guess there’s that. They fixed the problem so it shows ability to. Maybe the issue is “detect what parts we should do ourselves”? How do we detect censorship beyond blog posts like this? Can we diff various indices on a regular basis?
deleted 3y ago
[deleted]
lxe 3y ago
Thanks for the update! From your other thread: > Bing is our largest source of traditional web links. I think "traditional web links" is the main (and I think should be the only) product of a search engine, and it seems like you rely on Bing quite significantly to serve search results.
- blowski 3y ago
  I use DDG as my main search engineer. Many of my searches have hashbangs into other sites. On DDG landing pages, I often go straight to the right hand side of the page, with the info panel. Sometimes, I click on traditional web links, but with less frequency as the sites are so often spam.
- yegg 3y ago
  From lots of experience, I can tell you if you don't have these things (and others, this is an incomplete list just off the top of my head) showing up in the right places (often above "traditional web links") at the right times then you cannot get long-term search engine adoption from mainstream users: - maps - directions - local business listings - about boxes - basic facts - conversions - news - videos - images - shopping - sports - stocks - weather - flights - entertainment (e.g. cast) - etc.
  - Darthy 3y ago
    Gabriel, could you please clarify once and for all: for "traditional web links", how much do you rely on bing? The user consensus seems to be 100%. What is the official number from you? Thanks!
    - ncallaway 3y ago
      The previous statement seems to answer that reasonably well. > We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways I don't know that it's reasonable to demand that they provide you an exact percentage.
kikokikokiko 3y ago
It's turtles all the way down, I mean, it's Bing. If your basic source of information, the thing that you rely on to create any smoke and mirrors on top of it, i.e. the index, is basically Bing and just Bing, then you're just a Bing mirror. More and more I see myself using Yandex out of all things to get decent results. The extremes the web got in 2023 are bizarre, the "best" search engine in the world today may very well come from a dictatorship controlled country.
- photonerd 3y ago
  Either you’re very misinformed about what a mirror is, or you’re being extraordinarily disingenuous here. Bing is their initial and primary source of data for web links. Not the only one either, but the primary one. They maintain their own index from it. That’s not “a mirror” by any stretch of the imagination. What appears to have happened here is some form of replication bug where in a way yet to be determined a removal from the original data source was unintentionally replicated. Data synchronization is hard when you’re not directly mirroring data. This kind of bug crops up
  - kikokikokiko 3y ago
    If they're not just a mirror, it seems rather strange that when a specific website disappears from Bing's index, it also disappears from DDG. In my very common sense based view of the situation, if DDG really builds it's own "index" based on a variety of sources, then why did TechDirt completely disappeared from it's search results? DDG can argue all day about how much their "local and knowledge graph" is built in-house, yada yada yada. At the end of the day, a search engine must retrieve the most relevant websites for a given query. And nothing can be more relevant to "site:xxx.xxx.xxx" than the site itself. If it doesn't appear on DDG results, then it's not indexed. And then it clearly shows that DDG is nothing but a fancy mirror to Bing search.
    - photonerd 3y ago
      > it seems rather strange […] Not really. That’s the point. Data replication is hard, especially when you have to consistently update from the same data source. Bugs happen.
WheatMillington 3y ago
You can spin it any way you like, but the fact is evident: you rely on Bing to the extent that if Bing delists a site, that site will be completely sanitised from every single part of a DDG search, including all those dozens of modules you keep referring to.

FormerBandmate 3y ago

Why is Bing doing this? It feels like we’re burying the lede, a $2 trillion company isn’t showing sites that are critical of it on its supposedly unbiased search engine

criddell 3y ago
Does Microsoft actually claim Bing is unbiased?

pb7 3y ago

Regardless of what their CEO claims here every single time, if Bing cut off their access, DDG would be dead overnight. All the stuff they claim to do themselves is <1% of whatever value DDG provides. Lucky for them it’s just enough to be able to use terms like “largely” instead of “entirely”, “other sources” instead of “only source”, etc.

photonerd 3y ago
They wouldn’t die overnight at all. They would fast become outdated however, but that’s a different problem. One solvable with another source.