5 ms·

Why does DDG de-list things in the first place; isn't it in the interest of search engines to be as useful as possible to users, and thus maximise the results p

by nomilk 5y ago

Why does DDG de-list things in the first place; isn't it in the interest of search engines to be as useful as possible to users, and thus maximise the results provided? Also curious to know the extent to which Google de-lists things? Long term, perhaps a decentralised search engine could get around de-listing and provide a more reliable and rigorous search experience.

notatoad 5y ago
DDG uses the Bing index. If Bing de-lists something, then it disappears from DDG. Microsoft (and any other big company) has many competing interests other than just being helpful to users.
- nomilk 5y ago
  > [big companies have] many competing interests Are the main ones i) reducing competition and ii) managing their reputation? If so, the case for a decentralised search engine got stronger.
  - deleted 5y ago
    [deleted]
jbay808 5y ago
I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful. I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power. Not that centralized search engines are immune to this; they're almost as vulnerable (seeing as sponsored search results exist) but the maintainer at least has to balance that with the utility of the search engine overall, to prevent the search engine from falling out of favour. With a decentralized engine, parties that have deeply invested in manipulating the results will still want the engine to be popular too, but I'm not sure how you resolve the prisoners dilemma there as a whole.
- forgotmypw17 5y ago
  > I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful. > I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power. I would venture to say that a combination of allow-lists and block-lists from trusted parties, ranked using some kind of distributed web-of-trust system would work reasonably well.
  - KMag 5y ago
    Ages ago, I was a professional P2P developer, and I vaguely remember some of the research papers on P2P censorship-resistant reputation systems. Generally, you have public signing keys that sign ratings (say, -1.0 to 1.0) for both content and other raters. These signed ratings collections are then pushed into a distributed hash table. The basic idea is when you rate something, your client also looks up in the DHT other people who have rated the same content with similar ratings. Your client then pulls the latest ratings collections from those people, and computes the cosign distance between your ratings and their ratings (over the intersection of content that both of you have rated). Periodically, your client signs and publishes an updated ratings document, where the rating for other raters is the cosign distance. The cosign distance, the size of the ratings intersection set, and maybe some other factors go into deciding which raters get published out in your ratings update. When you query for the rating for a given piece of content, your client grabs the list of ratings for that content from the DHT. It then pulls the latest ratings published by those raters, computes cosign distance, and then does something similar to Djikstra's shortest-path algorithm to recursively search the DHT using these cosign distances as weights. In general, the DHT wouldn't have many signatures stored under the content's hash, but by recursively following the graph of other raters, your client hopefully finds other raters that rate things similarly to you and have rated this content. The path weight to a given rater is the product of cosign distances, and so by using a priority queue for querying, you get something close to a breadth-first search of the ratings graph. Once your client has accumulated enough weight of ratings for the given content, it stops and shows you the weighted average of the ratings (and maybe the weighted std. dev. is displayed as a confidence score to power users who have enabled it). Presumably, the UI for the ratings system maps 0 to 5 stars to 0.0 to 1.0 (probably not linearly, more likely the client locally keeps a histogram of the user's ratings and then maps the star rating back to a percentile rating), and the "spam" button rates the content as -1.0. The tricks come down to the metrics used for how the DHT decides priorities for cache eviction of the per-content ratings and also the per-rater ratings. You don't want spammers or other censors to be able to easily force cache eviction. Getting cache eviction metrics right is the key to having the system scale well while also preventing spammers/censors from evicting the most useful sets of ratings.
tdiff 5y ago
Banning Russian news outlets from search results was generally supported here on HN.
- gscott 5y ago
  There is no war just a special operation.
  - tonguez 5y ago
    we must condemn russias unprovoked genocide of neo nazis
    - suction 5y ago
      Wait, did Russians start committing suicide?
- ghostly_s 5y ago
  That's certainly not how I remember that thread going.
matheusmoreira 5y ago
Copyright industry pressure.
RedBeetDeadpool 5y ago
DDG CEO is clearly biased. Its no longer a neutral platform: https://twitter.com/yegg/status/1501716484761997318?s=20&t=9Z_0UQJohT3MCQ8zpFRe_g https://twitter.com/yegg/status/1501716484761997318?s=20&t=9... Before someone chimes in, yes I understand the humanitarian perspective. That's not my point. My point is that DDG is not neutral, and is politically biased.
- RedBeetDeadpool 5y ago
  BTW, if you wanted a biased platform, just use Google, its significantly better in every way. Ultimately what I'm getting at is: There's no market for DDG. Use Google for biased searches, and use other search engines that are not biased(which excludes DDG) for unbiased searches. What are the use cases for DDG?
  - pmoriarty 5y ago
    DDG respects privacy, or at least claims to. Google has nothing but contempt for it, and thrives on spying on you.
  - CameronNemo 5y ago
    Few people switched to DDG because of censorship, the main DDG selling point is privacy. All of their marketing stresses that aspect.
    - cinquemb 5y ago
      "All"[0] in the revision of historical examples sense. Unless one really thinks that filter bubbles have no overlap with censorship. Maybe i was only the few that cared more about the filter bubble angle and less the "we're selling bottled privacy™" angle [1] and am not interested now that `yegg` has clearly reneged upon the former [0] https://techcrunch.com/2011/06/20/duckduckgo-to-google-bing-users-escape-them-filter-bubbles/ https://techcrunch.com/2011/06/20/duckduckgo-to-google-bing-... [1] https://pictobar.tumblr.com/post/63785124046/the-banality-of-privacy-as-a-service https://pictobar.tumblr.com/post/63785124046/the-banality-of...
notriddle 5y ago
> Why does DDG de-list things in the first place I don’t think anyone wants to use a search engine that never delists anything. Ransomeware, Markov chain junk, plagiarism. A search engine that never delists anything is useless. The problem is when delisting is used against the end-user’s interests.
- anjbe 5y ago
  Early on, a major selling point of DuckDuckGo was that it filtered out SEO parked domain pages.
WhyNotHugo 5y ago
> Why does DDG de-list things in the first place [...] They use Bing as an index, and it was Bing who de-listed it.

by nomilk 5y ago

notatoad 5y ago

DDG uses the Bing index. If Bing de-lists something, then it disappears from DDG. Microsoft (and any other big company) has many competing interests other than just being helpful to users.

nomilk 5y ago
> [big companies have] many competing interests Are the main ones i) reducing competition and ii) managing their reputation? If so, the case for a decentralised search engine got stronger.
- deleted 5y ago
  [deleted]

jbay808 5y ago

I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful. I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power. Not that centralized search engines are immune to this; they're almost as vulnerable (seeing as sponsored search results exist) but the maintainer at least has to balance that with the utility of the search engine overall, to prevent the search engine from falling out of favour. With a decentralized engine, parties that have deeply invested in manipulating the results will still want the engine to be popular too, but I'm not sure how you resolve the prisoners dilemma there as a whole.

forgotmypw17 5y ago
> I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful. > I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power. I would venture to say that a combination of allow-lists and block-lists from trusted parties, ranked using some kind of distributed web-of-trust system would work reasonably well.
- KMag 5y ago
  Ages ago, I was a professional P2P developer, and I vaguely remember some of the research papers on P2P censorship-resistant reputation systems. Generally, you have public signing keys that sign ratings (say, -1.0 to 1.0) for both content and other raters. These signed ratings collections are then pushed into a distributed hash table. The basic idea is when you rate something, your client also looks up in the DHT other people who have rated the same content with similar ratings. Your client then pulls the latest ratings collections from those people, and computes the cosign distance between your ratings and their ratings (over the intersection of content that both of you have rated). Periodically, your client signs and publishes an updated ratings document, where the rating for other raters is the cosign distance. The cosign distance, the size of the ratings intersection set, and maybe some other factors go into deciding which raters get published out in your ratings update. When you query for the rating for a given piece of content, your client grabs the list of ratings for that content from the DHT. It then pulls the latest ratings published by those raters, computes cosign distance, and then does something similar to Djikstra's shortest-path algorithm to recursively search the DHT using these cosign distances as weights. In general, the DHT wouldn't have many signatures stored under the content's hash, but by recursively following the graph of other raters, your client hopefully finds other raters that rate things similarly to you and have rated this content. The path weight to a given rater is the product of cosign distances, and so by using a priority queue for querying, you get something close to a breadth-first search of the ratings graph. Once your client has accumulated enough weight of ratings for the given content, it stops and shows you the weighted average of the ratings (and maybe the weighted std. dev. is displayed as a confidence score to power users who have enabled it). Presumably, the UI for the ratings system maps 0 to 5 stars to 0.0 to 1.0 (probably not linearly, more likely the client locally keeps a histogram of the user's ratings and then maps the star rating back to a percentile rating), and the "spam" button rates the content as -1.0. The tricks come down to the metrics used for how the DHT decides priorities for cache eviction of the per-content ratings and also the per-rater ratings. You don't want spammers or other censors to be able to easily force cache eviction. Getting cache eviction metrics right is the key to having the system scale well while also preventing spammers/censors from evicting the most useful sets of ratings.

tdiff 5y ago

Banning Russian news outlets from search results was generally supported here on HN.

gscott 5y ago
There is no war just a special operation.
- tonguez 5y ago
  we must condemn russias unprovoked genocide of neo nazis
  - suction 5y ago
    Wait, did Russians start committing suicide?
ghostly_s 5y ago
That's certainly not how I remember that thread going.

matheusmoreira 5y ago

RedBeetDeadpool 5y ago

DDG CEO is clearly biased. Its no longer a neutral platform: https://twitter.com/yegg/status/1501716484761997318?s=20&t=9Z_0UQJohT3MCQ8zpFRe_g https://twitter.com/yegg/status/1501716484761997318?s=20&t=9... Before someone chimes in, yes I understand the humanitarian perspective. That's not my point. My point is that DDG is not neutral, and is politically biased.

RedBeetDeadpool 5y ago
BTW, if you wanted a biased platform, just use Google, its significantly better in every way. Ultimately what I'm getting at is: There's no market for DDG. Use Google for biased searches, and use other search engines that are not biased(which excludes DDG) for unbiased searches. What are the use cases for DDG?
- pmoriarty 5y ago
  DDG respects privacy, or at least claims to. Google has nothing but contempt for it, and thrives on spying on you.
- CameronNemo 5y ago
  Few people switched to DDG because of censorship, the main DDG selling point is privacy. All of their marketing stresses that aspect.
  - cinquemb 5y ago
    "All"[0] in the revision of historical examples sense. Unless one really thinks that filter bubbles have no overlap with censorship. Maybe i was only the few that cared more about the filter bubble angle and less the "we're selling bottled privacy™" angle [1] and am not interested now that `yegg` has clearly reneged upon the former [0] https://techcrunch.com/2011/06/20/duckduckgo-to-google-bing-users-escape-them-filter-bubbles/ https://techcrunch.com/2011/06/20/duckduckgo-to-google-bing-... [1] https://pictobar.tumblr.com/post/63785124046/the-banality-of-privacy-as-a-service https://pictobar.tumblr.com/post/63785124046/the-banality-of...

notriddle 5y ago

> Why does DDG de-list things in the first place I don’t think anyone wants to use a search engine that never delists anything. Ransomeware, Markov chain junk, plagiarism. A search engine that never delists anything is useless. The problem is when delisting is used against the end-user’s interests.

anjbe 5y ago
Early on, a major selling point of DuckDuckGo was that it filtered out SEO parked domain pages.

WhyNotHugo 5y ago

> Why does DDG de-list things in the first place [...] They use Bing as an index, and it was Bing who de-listed it.