4 ms·

Hi, I'm the author of this post and the CEO & Founder of DuckDuckGo. I'm happy to answer questions about it. I hope folks can read the whole thing but in short,

by yegg 4y ago

Hi, I'm the author of this post and the CEO & Founder of DuckDuckGo. I'm happy to answer questions about it. I hope folks can read the whole thing but in short, the post explains we're expanding our 3rd-Party Tracker Loading Protection to include blocking Microsoft trackers from loading, how this works with DuckDuckGo search ads, and a new help page that covers details on all our web tracking protections: https://help.duckduckgo.com/duckduckgo-help-pages/privacy/web-tracking-protections/ https://help.duckduckgo.com/duckduckgo-help-pages/privacy/we...

thankful69 4y ago
Are the search results being "curated" by its political nature or affiliation?
- yegg 4y ago
  They actually never were. We also recently put out a help page explaining how our news rankings work: https://help.duckduckgo.com/duckduckgo-help-pages/results/news-rankings/ https://help.duckduckgo.com/duckduckgo-help-pages/results/ne.... From that page: "when we apply our own ranking signals we do so in a strictly non-political manner, meaning we don’t evaluate or otherwise take into account any potential political bias or leanings of websites in our search result rankings."
  - thankful69 4y ago
    Cool, thanks for the clarification.
  - somenameforme 4y ago
    That entire page suggests extremely opaque behavior for a post about increasing transparency. Where can I find a list of which sites have been manually "downranked" and by how much? What are the "non-governmental" and "non-political" agencies declaring said sites to be whatever you're declaring them to be? Where are the links to the reports that informed your decisions?
    - yegg 4y ago
      This is a new help page, and so it is just a first step in more transparency on this issue. That said, what you're referencing in the page applies extremely rarely, currently to less than 0.1% of news websites, and it is expected to stay that way. So, you are unlikely to regularly encounter this ranking signal. And even then it's impact is relatively small, only a few slots on average -- from the page, "impacted sites are not moved so far down in the results that they are effectively removed."
      - somenameforme 4y ago
        So are you implying that you do have plans on revealing plans on revealing which sites you're "downranking"? And ideally precisely how they fit your criteria? I feel quite strongly about this because I, like I suspect many who use DDG, swapped over largely because I didn't really like the games Google was playing with search. As DDG goes down the same path I've moved on once again, but have been left with quite a sour taste in my mouth. I evangelized for your site for years, including on this site/account. Changes like this are the antithesis of everything that drew many people to DDG to begin with.
        yegg 4y ago
        Yes, we are constantly working towards more transparency in everything we do. More generally, from the new help page, "A search engine’s primary job is to rank results. In other words, search engines try to put results that most quickly and accurately answer the query on top." and "when we apply our own ranking signals we do so in a strictly non-political manner, meaning we don’t evaluate or otherwise take into account any potential political bias or leanings of websites in our search result rankings."
        riolu 4y ago
        Could you extrapolate on what you mean by "games"? I don't believe search engine bias is something that is an issue for Google or DDG. There is a lot of misinformation, it shouldn't be ranked near factual information, because humans are still bad at determining what is good information. Medical advice is a good example; this is a hyperbole, but if someone was choking and you needed to know the Heimlich, you probably don't want a result at the top for essential oils. Politically speaking, lots of people switched to DDG not for privacy, but because it never down ranked politically biased news like Google. I don't believe DDG's mission statement is to be the bastion of misinformation.
godelski 4y ago
Are there plans to do more search than wrapping Bing and Yandex?
- yegg 4y ago
  That isn't an accurate characterization of what we do now. We have on the order of a million lines of search code at this point and have a lot of talented people working them. As an example, mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. We don't get any local search content from Bing. Instead our local search content is a combination of our own indexes in partnership with Apple, TripAdvisor, and others. And then we have to further have a lot of code to know when and where to display that content on the page relative to other types of results, when to reject that content for not being relevant enough to display, and how to display parts of it that are relevant enough. In addition, we currently do not use Yandex for search content.
  - lapcat 4y ago
    What can you do to prevent sites from getting removed from DuckDuckGo as a side effect of Bing mistakenly deindexing the sites? https://www.jessesquires.com/blog/2022/07/25/my-website-disappeared-from-bing-and-duckduckgo-part-2/ https://www.jessesquires.com/blog/2022/07/25/my-website-disa... https://lapcatsoftware.com/articles/bing.html https://lapcatsoftware.com/articles/bing.html https://io.bikegremlin.com/28530/microsoft-bing-serp-gone-overnight/ https://io.bikegremlin.com/28530/microsoft-bing-serp-gone-ov... https://techcommunity.microsoft.com/t5/microsoft-bing/help-why-my-site-has-been-removed-from-bing-s-search-results/m-p/2906441 https://techcommunity.microsoft.com/t5/microsoft-bing/help-w...
    - yegg 4y ago
      Best would be to report them to us and we can investigate. I see these ones coming up though right now: https://duckduckgo.com/?q=jessesquires.com https://duckduckgo.com/?q=jessesquires.com, https://duckduckgo.com/?q=lapcatsoftware.com https://duckduckgo.com/?q=lapcatsoftware.com, https://duckduckgo.com/?q=bike.bikegremlin.com https://duckduckgo.com/?q=bike.bikegremlin.com
whollythrowaway 4y ago
How did you guys get into so much jeopardy over the Microsoft partnership in the first place? How did Startpage figure all this out, apparently with even better privacy protections, for longer, and with Google's Index - the best one? What is going on?
- yegg 4y ago
  This is actually about our browser and web tracking protections within it around third-party scripts on other websites. From the post: "Microsoft scripts were never embedded in our search engine or apps, which do not track you. Websites insert these scripts for their own purposes, and so they never sent any information to DuckDuckGo."
  - whollythrowaway 4y ago
    If you expand the boundary of end user privacy services to "browser extensions," isn't the most intellectually honest answer to have people use an EasyList-adjacent accelerated solution, like uBlock Origin or AdGuard on mobile? That is certainly the consensus here. Is anything in the blog post an improvement compared to that offering? That's why I'm asking about Startpage. Why do you think DuckDuckGo uniquely got put into a position of jeopardy - something more than scrutiny - when others did not?
    - yegg 4y ago
      Yes, please check out the comprehensive help page referenced for the list of web tracking protections we offer by default across platforms, some of which are not offered by related extensions, and most all of which are not offered by most browsers by default: https://help.duckduckgo.com/duckduckgo-help-pages/privacy/web-tracking-protections/ https://help.duckduckgo.com/duckduckgo-help-pages/privacy/we.... This is the full list for reference/comparison (platform support can vary and in some cases is impossible, but working hard to get them on all platforms -- the help page has all the details): 3rd-Party Tracker Loading Protection, 3rd-Party Cookie Protection, 1st-Party Cookie Protection, CNAME Cloaking Protection, Fingerprinting Protection, Smarter Encryption (HTTPS Upgrading), Link Tracking Protection, Referrer Tracking Protection, Embedded Social Content Tracking Protection, Google AMP Protection, Google Topics Protection, Google FLEDGE Protection, Surrogates, The Fire Button, Cookie Consent Pop-Up Management, Global Privacy Control (GPC). More fundamentally though our web tracking protections are built upon a data set (we call Tracker Radar) that is frequently updated based on web crawling (we call our Tracker Radar Detector), which offers a much more comprehensive picture of third-party web tracking of which to base lists and evolving protections relative to solely community maintained lists. Both of these are open source on github. We also have an analogous data set for our HTTPS upgrade list (we call Smarter Encryption), updated daily based on continuous crawling, which is also open source.
lysergia 4y ago
Gabriel, whilst you’re answering questions here, I have one slightly tangential question and it’s as follows: why does DDG not have a warrant canary? I’m skeptical of using the service since it’s a US based service and the US doesn’t have a good track record when it comes to privacy. For this reason I only ever use DDG combined with an anonymous mixer service like Tor. It would be trivial for bad actors to be placed in your data center and tie specific IPs to particular searches, no?
- yegg 4y ago
  Warrant canaries are problematic since they can't be trusted themselves due to possible confidentiality restrictions, can be easily misinterpreted, and can also be counter-productive to an in-process court proceeding if you are in an actual situation that would involve triggering one. Thankfully we haven't ever been in any situation like this because we simply don't collect any search histories or meta data that ties searches to individuals (like IP addresses). And our searches are end-to-end encrypted so network onlookers cannot see them, even if listening in on the network.
calculatte 4y ago
Congratulations on throwing yourself into the dumpster of internet search. I appreciate your efforts to push people toward using Brave Search, Qwant, and Presearch instead of DDG's censored results. Keep "curating" yourself out of the ecosystem.
- yegg 4y ago
  We don’t censor results. Unless legally prohibited, you should find all media outlets in our results. Related, we recently published another help page explaining how our news rankings work: https://help.duckduckgo.com/duckduckgo-help-pages/results/news-rankings/ https://help.duckduckgo.com/duckduckgo-help-pages/results/ne....
  - groffee 4y ago
    Sure you do. Remember tank man? [0] You don't index shit yourselves so when Bing censors something you do too. [0] https://news.ycombinator.com/item?id=27394925 https://news.ycombinator.com/item?id=27394925
    - yegg 4y ago
      We do index many things (see related comment at https://news.ycombinator.com/item?id=32360874 https://news.ycombinator.com/item?id=32360874). We do not remove any results ourselves for political purposes and in fact we have been banned in China for many years for that very reason. What you're referring to was a temporary bug in our image search results from Bing that they promptly fixed. If they hadn't fixed it promptly then we would have taken further action.
      - barbacoa 4y ago
        Just now I went to RT.com and searched various article titles verbatim on DDG, they came up near the bottom half of results. I repeated this test with WSJ and found the link as the #1 result. From this it wouldn't be unreasonable to believe that either you or the sources you pull from are engaging in arbitrary website throttling/downranking. I hear you when you say you don't remove content, but do you, or to what extent do you throttle/downrank websites?
pessimizer 4y ago
Unrelated (although not completely), on your news search, the msn.com reprint of an article usually shows up far more often than the original article. Is this on the list of things to get to?
- yegg 4y ago
  Yes.
zagrebian 4y ago
Isn’t there a way to measure ad effectiveness that doesn’t require making exceptions for scripts from tracking domains? Browsers should implement a standard API for this. I trust my browser. I do not trust websites.

by yegg 4y ago

thankful69 4y ago

Are the search results being "curated" by its political nature or affiliation?

yegg 4y ago
They actually never were. We also recently put out a help page explaining how our news rankings work: https://help.duckduckgo.com/duckduckgo-help-pages/results/news-rankings/ https://help.duckduckgo.com/duckduckgo-help-pages/results/ne.... From that page: "when we apply our own ranking signals we do so in a strictly non-political manner, meaning we don’t evaluate or otherwise take into account any potential political bias or leanings of websites in our search result rankings."
- thankful69 4y ago
  Cool, thanks for the clarification.
- somenameforme 4y ago
  That entire page suggests extremely opaque behavior for a post about increasing transparency. Where can I find a list of which sites have been manually "downranked" and by how much? What are the "non-governmental" and "non-political" agencies declaring said sites to be whatever you're declaring them to be? Where are the links to the reports that informed your decisions?
  - yegg 4y ago
    This is a new help page, and so it is just a first step in more transparency on this issue. That said, what you're referencing in the page applies extremely rarely, currently to less than 0.1% of news websites, and it is expected to stay that way. So, you are unlikely to regularly encounter this ranking signal. And even then it's impact is relatively small, only a few slots on average -- from the page, "impacted sites are not moved so far down in the results that they are effectively removed."
    - somenameforme 4y ago
      So are you implying that you do have plans on revealing plans on revealing which sites you're "downranking"? And ideally precisely how they fit your criteria? I feel quite strongly about this because I, like I suspect many who use DDG, swapped over largely because I didn't really like the games Google was playing with search. As DDG goes down the same path I've moved on once again, but have been left with quite a sour taste in my mouth. I evangelized for your site for years, including on this site/account. Changes like this are the antithesis of everything that drew many people to DDG to begin with.
      - yegg 4y ago
        Yes, we are constantly working towards more transparency in everything we do. More generally, from the new help page, "A search engine’s primary job is to rank results. In other words, search engines try to put results that most quickly and accurately answer the query on top." and "when we apply our own ranking signals we do so in a strictly non-political manner, meaning we don’t evaluate or otherwise take into account any potential political bias or leanings of websites in our search result rankings."
      - riolu 4y ago
        Could you extrapolate on what you mean by "games"? I don't believe search engine bias is something that is an issue for Google or DDG. There is a lot of misinformation, it shouldn't be ranked near factual information, because humans are still bad at determining what is good information. Medical advice is a good example; this is a hyperbole, but if someone was choking and you needed to know the Heimlich, you probably don't want a result at the top for essential oils. Politically speaking, lots of people switched to DDG not for privacy, but because it never down ranked politically biased news like Google. I don't believe DDG's mission statement is to be the bastion of misinformation.

godelski 4y ago

Are there plans to do more search than wrapping Bing and Yandex?

yegg 4y ago
That isn't an accurate characterization of what we do now. We have on the order of a million lines of search code at this point and have a lot of talented people working them. As an example, mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. We don't get any local search content from Bing. Instead our local search content is a combination of our own indexes in partnership with Apple, TripAdvisor, and others. And then we have to further have a lot of code to know when and where to display that content on the page relative to other types of results, when to reject that content for not being relevant enough to display, and how to display parts of it that are relevant enough. In addition, we currently do not use Yandex for search content.
- lapcat 4y ago
  What can you do to prevent sites from getting removed from DuckDuckGo as a side effect of Bing mistakenly deindexing the sites? https://www.jessesquires.com/blog/2022/07/25/my-website-disappeared-from-bing-and-duckduckgo-part-2/ https://www.jessesquires.com/blog/2022/07/25/my-website-disa... https://lapcatsoftware.com/articles/bing.html https://lapcatsoftware.com/articles/bing.html https://io.bikegremlin.com/28530/microsoft-bing-serp-gone-overnight/ https://io.bikegremlin.com/28530/microsoft-bing-serp-gone-ov... https://techcommunity.microsoft.com/t5/microsoft-bing/help-why-my-site-has-been-removed-from-bing-s-search-results/m-p/2906441 https://techcommunity.microsoft.com/t5/microsoft-bing/help-w...
  - yegg 4y ago
    Best would be to report them to us and we can investigate. I see these ones coming up though right now: https://duckduckgo.com/?q=jessesquires.com https://duckduckgo.com/?q=jessesquires.com, https://duckduckgo.com/?q=lapcatsoftware.com https://duckduckgo.com/?q=lapcatsoftware.com, https://duckduckgo.com/?q=bike.bikegremlin.com https://duckduckgo.com/?q=bike.bikegremlin.com

whollythrowaway 4y ago

How did you guys get into so much jeopardy over the Microsoft partnership in the first place? How did Startpage figure all this out, apparently with even better privacy protections, for longer, and with Google's Index - the best one? What is going on?

yegg 4y ago
This is actually about our browser and web tracking protections within it around third-party scripts on other websites. From the post: "Microsoft scripts were never embedded in our search engine or apps, which do not track you. Websites insert these scripts for their own purposes, and so they never sent any information to DuckDuckGo."
- whollythrowaway 4y ago
  If you expand the boundary of end user privacy services to "browser extensions," isn't the most intellectually honest answer to have people use an EasyList-adjacent accelerated solution, like uBlock Origin or AdGuard on mobile? That is certainly the consensus here. Is anything in the blog post an improvement compared to that offering? That's why I'm asking about Startpage. Why do you think DuckDuckGo uniquely got put into a position of jeopardy - something more than scrutiny - when others did not?
  - yegg 4y ago
    Yes, please check out the comprehensive help page referenced for the list of web tracking protections we offer by default across platforms, some of which are not offered by related extensions, and most all of which are not offered by most browsers by default: https://help.duckduckgo.com/duckduckgo-help-pages/privacy/web-tracking-protections/ https://help.duckduckgo.com/duckduckgo-help-pages/privacy/we.... This is the full list for reference/comparison (platform support can vary and in some cases is impossible, but working hard to get them on all platforms -- the help page has all the details): 3rd-Party Tracker Loading Protection, 3rd-Party Cookie Protection, 1st-Party Cookie Protection, CNAME Cloaking Protection, Fingerprinting Protection, Smarter Encryption (HTTPS Upgrading), Link Tracking Protection, Referrer Tracking Protection, Embedded Social Content Tracking Protection, Google AMP Protection, Google Topics Protection, Google FLEDGE Protection, Surrogates, The Fire Button, Cookie Consent Pop-Up Management, Global Privacy Control (GPC). More fundamentally though our web tracking protections are built upon a data set (we call Tracker Radar) that is frequently updated based on web crawling (we call our Tracker Radar Detector), which offers a much more comprehensive picture of third-party web tracking of which to base lists and evolving protections relative to solely community maintained lists. Both of these are open source on github. We also have an analogous data set for our HTTPS upgrade list (we call Smarter Encryption), updated daily based on continuous crawling, which is also open source.

lysergia 4y ago

Gabriel, whilst you’re answering questions here, I have one slightly tangential question and it’s as follows: why does DDG not have a warrant canary? I’m skeptical of using the service since it’s a US based service and the US doesn’t have a good track record when it comes to privacy. For this reason I only ever use DDG combined with an anonymous mixer service like Tor. It would be trivial for bad actors to be placed in your data center and tie specific IPs to particular searches, no?

yegg 4y ago
Warrant canaries are problematic since they can't be trusted themselves due to possible confidentiality restrictions, can be easily misinterpreted, and can also be counter-productive to an in-process court proceeding if you are in an actual situation that would involve triggering one. Thankfully we haven't ever been in any situation like this because we simply don't collect any search histories or meta data that ties searches to individuals (like IP addresses). And our searches are end-to-end encrypted so network onlookers cannot see them, even if listening in on the network.

calculatte 4y ago

Congratulations on throwing yourself into the dumpster of internet search. I appreciate your efforts to push people toward using Brave Search, Qwant, and Presearch instead of DDG's censored results. Keep "curating" yourself out of the ecosystem.

yegg 4y ago
We don’t censor results. Unless legally prohibited, you should find all media outlets in our results. Related, we recently published another help page explaining how our news rankings work: https://help.duckduckgo.com/duckduckgo-help-pages/results/news-rankings/ https://help.duckduckgo.com/duckduckgo-help-pages/results/ne....
- groffee 4y ago
  Sure you do. Remember tank man? [0] You don't index shit yourselves so when Bing censors something you do too. [0] https://news.ycombinator.com/item?id=27394925 https://news.ycombinator.com/item?id=27394925
  - yegg 4y ago
    We do index many things (see related comment at https://news.ycombinator.com/item?id=32360874 https://news.ycombinator.com/item?id=32360874). We do not remove any results ourselves for political purposes and in fact we have been banned in China for many years for that very reason. What you're referring to was a temporary bug in our image search results from Bing that they promptly fixed. If they hadn't fixed it promptly then we would have taken further action.
    - barbacoa 4y ago
      Just now I went to RT.com and searched various article titles verbatim on DDG, they came up near the bottom half of results. I repeated this test with WSJ and found the link as the #1 result. From this it wouldn't be unreasonable to believe that either you or the sources you pull from are engaging in arbitrary website throttling/downranking. I hear you when you say you don't remove content, but do you, or to what extent do you throttle/downrank websites?

pessimizer 4y ago

Unrelated (although not completely), on your news search, the msn.com reprint of an article usually shows up far more often than the original article. Is this on the list of things to get to?

yegg 4y ago
Yes.

zagrebian 4y ago

Isn’t there a way to measure ad effectiveness that doesn’t require making exceptions for scripts from tracking domains? Browsers should implement a standard API for this. I trust my browser. I do not trust websites.