31 ms·

DuckDuckGo Removes Pirate Sites and YouTube-DL from Its Search Results

864 points by TangerineDream 5y ago

zanethomas 5y ago
duckduckgone
- einpoklum 5y ago
  Damn it! You beat me to this quip! Here's your damn +1. Grrr...
cpach 5y ago
I don’t condone the actions of the Russian government; however: If you want better results from pirate sites, try Yandex.com. Quite good for finding torrents.
- Jerry2 5y ago
  >Quite good for finding torrents. That's because there's no DMCA law there and they're not taking down thousands upon thousands of pages like Google is forced to do every day. [1] [1] https://www.lumendatabase.org/ https://www.lumendatabase.org/
  - codedokode 5y ago
    Yandex has an agreement with copyright holders and removes sites with pirated content from results. But those are Russian copyright holders and they mostly care about content in Russian. Also Yandex removes sites that are banned in Russia, for example BBC in Russian or Navalny's site.
- probably_wrong 5y ago
  I used to be able to find MP3s with Google, until one day it simply stopped working. Then I moved to Baidu, which worked for a while. I now use DDG to find movies, and yet looks like the curtain may be closing on that one too. I fear it might only be a matter of time before Yandex stops working too.
  - Markoff 5y ago
    there are telegram bots for it though, personally I don't use them but people swear on them btw Baidu is extremely slow due to GFW
    - itsTyrion 5y ago
      true that. I tried to look something up on Baidu once (from Germany) and gave up after it loaded for a full minute
- draugadrotten 5y ago
  Russian government has always appreciated https://en.wikipedia.org/wiki/Kompromat https://en.wikipedia.org/wiki/Kompromat so be careful what you search for.
- seanw444 5y ago
  I love how we have to preface our approval of something foreign with "now I'm not a Nazi, but...'
  - cpach 5y ago
    The obvious reason I used that disclaimer was the fact that Russia’s government decided to invade Ukraine.
- Markoff 5y ago
  why would you look for torrents in search engine? you will anyway find most of them dead you are much better off with Knaben on EU domain, same results can be achieved with add-ons in qbit, then there are special software not worth my time (Jackett) but I agree I find myself using yandex much more nowadays as alternative
  - cpach 5y ago
    Yandex made it possible for me to find extremely rare movies that are not to be found elsewhere. So I have found it very useful.
mdaniel 5y ago
As for the YouTube-DL part specifically, I'm pretty sure that's a side-effect of them being dependent upon Bing for the actual index: https://www.bing.com/search?q=site%3Ayoutube-dl.org https://www.bing.com/search?q=site%3Ayoutube-dl.org Also, _come on_; if someone gives up after one search term, which also includes advanced site-restricting syntax, there's no way they'd be able to operate youtube-dl anyway: https://duckduckgo.com/?q=youtube-dl&ia=web https://duckduckgo.com/?q=youtube-dl&ia=web
- muhammadusman 5y ago
  Yeah idk why someone would type in the full URL of a site and not just the term they're looking for in a search engine...what a dumb way to test that on their part.
  - miloignis 5y ago
    The point isn't that you can't reach the website by typing in the search term, the point is that they've deindexed the site. If you search for youtube-dl, neither youtube-dl.org nor yt-dl.org come up in the results, as far as I can see. You can still access it from the results via other sites, like github and wikipedia (which even makes the real website pop up in the info square, funnily enough), but the search results themselves do not contain any links to the main website.
  - daxuak 5y ago
    What if I want to search what sites have mentioned those urls?
    - lolinder 5y ago
      That's still the wrong syntax.
  - ykonstant 5y ago
    Absolutely! Why on earth would you test a dumb way of doing things on a user-facing service?! slams hand on desk Irrelevant!
deleted 5y ago
[deleted]
ravenstine 5y ago
I'm just waiting for the day they announce an NFT or a "trusted partners" program with establishment media entities. Come on, DDG, you've come this far, so truly jump the shark for our amusement.
- bouncycastle 5y ago
  Which would be ironic since Nfts cannot be censored (they exist on a decentralized censorship resistant database)
mikece 5y ago
I thought yt-dlp was recommended over youtube-dl these days. Any chance DDG suppressing a less-preferred program?
- ajyotirmay 5y ago
  It's a search engine, not a popularity ranking system.
Vladimof 5y ago
- thematrixturtle 5y ago
  Google pushed the envelope on how silly the name for a major company can be, but DuckDuckGo is unlikely to ever hit mass market success with that name. (Or, more importantly, the current implementation.)
  - Vladimof 5y ago
    Google even gave duck.com to them, for free, I think I used duckduckgo on and off for probably a year ... might as well use Bing if you like it
    - iinnPP 5y ago
      I don't think Google does anything for free. Sure provides a good reason not to use it.
  - gigglesupstairs 5y ago
    It also owns www.duck.com. Not sure why they’re still sticking with DuckDuckGo.
- NoImmatureAdHom 5y ago
  tbh I find it to be better than Google
  - Vladimof 5y ago
    tbh I find that all search engines are worst then Google was 10 years ago (including Google)
    - NoImmatureAdHom 5y ago
      Agreed. Gimme goddamn booleans that work!
nmilo 5y ago
Youtube-dl's site is yt-dl.org, not youtube-dl.org. Not surprised the second one doesn't show up. Not sure why this blogspam is here on HN either.
- miloignis 5y ago
  yt-dl.org also seems not to appear: https://duckduckgo.com/?q=site%3Ayt-dl.org&t=h_&ia=web https://duckduckgo.com/?q=site%3Ayt-dl.org&t=h_&ia=web
- mdaniel 5y ago
  They are both bing suppressed, and as best I can tell are synonymous $ dig +short youtube-dl.org. 95.143.172.170 $ dig +short yt-dl.org. 95.143.172.170 as for the blogspam, I'm sure everyone loves a good free speech suppression story, but yeah, I wish HN offered downvotes on submissions, too
- syoc 5y ago
  I agree that was not a high quality post from TorrenFreak. I do however also find the blogspam label to be unfair. TF has pushed some important agendas over the years.
slig 5y ago
So much for a Google alternative.
Apreche 5y ago
Someone should make a search engine that only indexes sites that Google and DuckDuckgo do not index. It would serve sort of the same purpose as like, lists of banned books.
- bufferoverflow 5y ago
  It will be a lot of spam with a bit of useful sites sprinkled randomly.
- jfengel 5y ago
  You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links? Yeah, I guess that would probably be useful for something. Academic research into just how awful people are, perhaps. And, of course, as a blackhole list. Just make sure you get a bulk discount on disk space. Such a list would be huge.
  - theknocker 5y ago
  - darkteflon 5y ago
    To be fair, “SEO-driven gibberish with back-links” is an apt description of Google’s actual results these days.
  - Brian_K_White 5y ago
    "You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links?" Yes, exactly, I want exactly whatever I search for, including that if that's what I search for. If I search for something which exists, then I want the knowledge of that existence. The value judgement is no one else's business unless I search for "X but only where approved by any governments or companies with any political or financial interest in controlling what I know." When it turns out that I would never choose to search for that, that is the proof that it's wrong to be doing it.
    - 5y ago
twoxproblematic 5y ago
eimrine 5y ago
Who really needs a search engine nowadays? The google or ddg or whatever is just a search aggregator for making no need to open Wikipedia for searching who is Franko then open weather.com for figuring out the weather next weekend then open SO for quitting vim and so on. I can not remember when I really searched something general last time which is impossible to find via single-site search. And of course free porn videos and other pirated stuff and cool-hacker-utils is never been searched using government-friendly search businesses.
- jjulius 5y ago
  It's cool that you've had enough experience to just know where everything that you need is. Not everybody has the same experience with finding things on the internet as you, nor is the average internet user anywhere near as technically-minded as you.
- drexlspivey 5y ago
  > Who really needs a search engine nowadays? Apparently a lot of people since google processes 6B searches every day?
  - thematrixturtle 5y ago
    I have no idea what the actual number is, but that sounds low. The average HNer probably hits three digits every day easily.
  - eimrine 5y ago
    6B showings of ad-loaded webpage does not mean 6B successful results shown. And my claim is that most of successful google searches is just a wrong choice of a search engine.
- BlackestNight 5y ago
kingcharles 5y ago
What about archive.org? Will they delist that too? That definitely contains a lot of copyright-infringing data. Where will this end? Pirate Bay actually contains a lot of non-infringing items too. Who watches the watchers?
- yjftsjthsd-h 5y ago
  > Pirate Bay actually contains a lot of non-infringing items too. Wait, what have they got? Actual Linux ISOs?
  - robbedpeter 5y ago
    And public domain content, and content licensed for free distribution, and abandoned software. The good stuff is there too, but the legit free stuff isn't all bad.
- KSPAtlas 5y ago
  You can literally get Windows keys on archive.org, not that I use Windows anyways
  - ajyotirmay 5y ago
    10yo me would have loved this information. Linux FTW
walrus01 5y ago
isn't duckduckgo just a skin on top of bing?
- HigherPlain 5y ago
  It's Bing.
- bduerst 5y ago
  They claim to have results from over "400 sources" but then don't list them.
anm89 5y ago
And I have removed duck duck go as my default browser
- anm89 5y ago
  *search engine
  - ajyotirmay 5y ago
    What are you switching to?
thematrixturtle 5y ago
I suspect this is largely due to DDG using Bing under the hood, which has led to similar weirdness in the past, eg all major porn sites disappearing from the results in Singapore (while Google still showed them, mind you!). Still super disappointing though, and yet another reason why trying to build a better search engine on top of someone else's tech is a non-starter.
- KMag 5y ago
  Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it on over the visible web. You need to bootstrap somehow form an existing index if you want to get anywhere in any reasonable time. If there's a market for alternative search engines out there, AWS is already serving so much of the web that Amazon should really provide crawling and basic indexing (for a fee, of course) so it's done once for everyone, with Amazon Lambda processing to allow search engines to customize their indexing. I'm not sure if it would make them much money, but it makes more sense to start up a search engine on AWS using Amazon's crawl and basic indexing vs. using a search engine competitor's crawl and indexing.
  - post-it 5y ago
    That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO. Awful for competition and whatnot.
    - KMag 5y ago
      Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition. As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.
barnabee 5y ago
I don’t need these results but I don’t want to support this behaviour/trend. Can anyone recommend a decent non-Google alternative.
- ColinHayhurst 5y ago
  Your choices are limited, and very few independent indexes. Most options are Bing alternatives: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
  - mdaniel 5y ago
    That was fascinating, thanks so much for sharing it
- theandrewbailey 5y ago
  https://search.brave.com/ https://search.brave.com/
- nabeards 5y ago
  https://presearch.org https://presearch.org
  - xigoi 5y ago
    Cryptocurrency? No, thanks.
- lolinder 5y ago
  I've been using Kagi (https://kagi.com https://kagi.com) for a few months and it's been great. It's free while in beta but will be funded by subscription fees afterward. The killer feature for me is the ability to up-rank and down-rank sites in my personal results, which has been really helpful for curating my tech-related searches. It allows me to quickly find high-quality docs and eliminate the garbage.
  - 5y ago
babypuncher 5y ago
What is with this blogspam? YouTube-DL comes up just fine in search results (https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web). So does Pirate Bay (https://duckduckgo.com/?q=pirate+bay&t=h_&ia=web https://duckduckgo.com/?q=pirate+bay&t=h_&ia=web). The headline is outright false.
- Flollop 5y ago
  the complaint in the article is around the serach engine's "search within site" feature. Compare the following two: - https://duckduckgo.com/?q=site%3Athepiratebay.org&t=h_&ia=web https://duckduckgo.com/?q=site%3Athepiratebay.org&t=h_&ia=we... - https://duckduckgo.com/?q=site%3Aycombinator.com&t=h_&ia=web https://duckduckgo.com/?q=site%3Aycombinator.com&t=h_&ia=web
- miloignis 5y ago
  The YouTube-DL website notably does not come up in those search results, it's their GitHub page that does. The website does appear in the sidebar, because it pulls the sidebar data from Wikipedia, and YouTube-DL's Wikipedia page does come up. Note that if you do a google search (https://www.google.com/search?q=youtube-dl https://www.google.com/search?q=youtube-dl) their website is the first result.
  - gigglesupstairs 5y ago
    Both the website and the GitHub are top two results at my end, in India.
    - miloignis 5y ago
      Interesting, mine are github, videohelp.com, and then ottverse. I imagine that adds credence to the theory that it's because of Bing's index removing them - maybe they use a different index in India?
    - omgmajk 5y ago
      Their main website is not coming up here (sweden). Only the github and some random fringe-sites, obvious scams and readthedocs.
rolph 5y ago
a search using; anyterm :: torrent still returns alot of results for me sofar. and so does youtube-dl; top result is : github.com/ytdl-org
- ajyotirmay 5y ago
  That's the github link, not youtube-dl website :)
  - rolph 5y ago
    I can still reach it yt-dl.org and as youve noticed the repo is still part of search results as well.
srvmshr 5y ago
Serious Question: How is the DDG search structured? Is it a cosmetic skin over Bing, or is it aggregating from other sites like Yahoo, ecosia etc additionally? If it is just Bing under the hood, how does it exist as business entity. I am sure MS will take some action to consolidate their search share rather than seeing splintered.
- marpstar 5y ago
  They aggregate from "over 400 sources" and run their own crawler: https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
  - bduerst 5y ago
    Do they list their "sources"? It reads like they are saying a website = source, and the hyperlink for the sources is dead.
  - cato_the_elder 5y ago
    That's just their corporate propaganda. For the most part, it's Bing with some additional features. Their crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [1] [1]: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html#fn:2 https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
    - etataetaet 5y ago
      To be fair the instant answers are usually quite good!
- cato_the_elder 5y ago
  > If it is just Bing under the hood, how does it exist as business entity. They provide an alternative branding, targeted at "privacy-aware" users and hipsters.
- HigherPlain 5y ago
deleted 5y ago
[deleted]
AdmiralAsshat 5y ago
I've started hearing ads for DuckDuckGo on NPR, of all places. Could be completely coincidental, but I find the timing suspicious. Do you think they're gearing up for an IPO?
- derekbaker783 5y ago
  I think they've advertised there for a while.
- ronsor 5y ago
  I've seen DuckDuckGo ads on network TV.
zekrioca 5y ago
At least when querying "youtube-dl" it shows a small snippet to the right where one can see the URL..
Kiro 5y ago
Funny how the narrative on DDG has changed. I used to get downvoted to oblivion for merely mentioning they used Bing under the hood. Where are all the people that used to defend them so vigorously? I wouldn't be surprised if it's the same people now hating on them.
- tibyat 5y ago
- qiskit 5y ago
  > I wouldn't be surprised if it's the same people now hating on them. It seems like those people are still here defending ddg. They are even attacking the blog. Weird. I use ddg as my default search and hope they are "privacy-first" as they claim ( thought I have my doubts ). At this point what other option is there? DDG search is noticably censored since they rely on bing. It's obvious to anyone who uses ddg regularly. Not sure why people here are making excuses for ddg or pretending otherwise. What I would give to get the google of old. It's sad looking back on what google used to be. All the freedom and optimism of the 2000s is definitely gone.
  - seanw444 5y ago
    Searx. Use Searx. Does your search engine of choice fail to get results that the others pick up? Searx grabs all of them and compiles them together. Don't miss another search result! Also, proxying increases privacy. Also, it's open source and self-hostable.
    - prmoustache 5y ago
      Being open source doesn't mean the people running the public searx instances aren't tracking you. Self-hosting makes you lose all the benefits of proxying. In the end be it ddg or searx you can only rely on some third-party trust.
      - seanw444 5y ago
autoexec 5y ago
It's amazing how much collateral damage is caused by our horrible copyright laws. Mostly just so the MPA/RIAA can protect their roles as gatekeepers of what we're allowed to see and hear. They can put enormous pressure on even the wealthiest and most powerful companies to act as copyright police on their behalf. Even Google is afraid of them. ISPs are forced to spend huge amounts of time and money working for them. Now duckduckgo is being strong armed into doing a bunch of free work for them too? Maintaining lists of websites and domains to block and removing links to even non-infringing material like youtube-dl just to keep from being sued into the ground. I don't know what it'll take to rein in these guys, but I doubt the courts will be the ones to do it. So far courts seem fine with the idea that ISPs must permanently ban users from their service over nothing but repeated unsubstantiated claims of infringements which is an insane amount of power to give any industry. Has any US politician ever run on a platform that includes copyright reform?
- rglullis 5y ago
  Lawrence Lessig
  - autoexec 5y ago
    I forgot about that! What a missed opportunity that was for this country. He's exactly the sort of person we need to see more of in politics.
    - perardi 5y ago
      His campaign was much more about electoral reform than copyright reform. https://en.wikipedia.org/wiki/Lawrence_Lessig_2016_presidential_campaign https://en.wikipedia.org/wiki/Lawrence_Lessig_2016_president... Which is broadly more important than copyright reform…and broadly Americans don’t seem to particularly care about either of them all that much.
      - wussboy 5y ago
        Either would have been a win as far as I’m concerned.
teekert 5y ago
Hmm this will affect me, I always type tpb in ddg to go there. I’m also happy my provider is small and freedom loving so it isn’t blocked (neither is RT for that matter).
JohnWhigham 5y ago
What is going on with this company in the past several months? Completely destroying any goodwill they had
zodzedzi 5y ago
I use DDG as my default search engine, along with NoScript in the browser. Often when I visit a new website, I peruse the (long) list of domains that the site is trying to pull javascripts from. I keep most of those source sites in UNTRUSTED status (including some of the big names in search/ads/etc). But I've always had DDG in the TRUSTED category because I had only seen its javascript before on the main DDG website. (Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com"). But recently I started noticing some websites pulling javascript from DDG (I don't remember which sites). So now I was wondering if DDG is getting into the tracking business, since they're now having their javascripts load from third party sites. Obviously this is anecdotal. But does anyone know if they are indeed beginning to track?
- stjohnswarts 5y ago
  Can you give us a list (or partial list) of sites that are pulling scripts from duckduckgo? We can look at what they're trying to do.
  - zodzedzi 5y ago
    I don't remember the sites; I'll try to find them again, and will share here if I do. I remember seeing 3 sites within an hour, and deciding to change the DDG setting to TEMP:TRUSTED afterwards.
    - autoexec 5y ago
      The nice thing about DDG is that if you're willing to sacrifice some functionality it can still be used without JS at all (which is how I use it)
- z3c0 5y ago
  > Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com" I was under the impression that the custom option allowed this. Am I misunderstanding the point of this option?
derevaunseraun 5y ago
Question: does anyone actually know what DDG does with user data? Like they market themselves as a "privacy respecting" search engine, but how much of this is truth? I'd imagine there's good money in convincing people they have privacy because then they'll provide more interesting data. Has the company ever been audited? Why should they be trusted to not compromise user privacy? Imo at least Google is honest: you know when you use their products as intended you have no privacy, and they don't try to hide this Edit: since DDG isn't open source like searx, how do we know there is ANY truth to their marketing claims? Edit: Just for accuracy, the browser extensions are open source. But as far as I know, the actual search engine isn't Edit: They made over 100 million in 2020. They clearly can (and should) get an independent audit. It's shocking that they haven't had a single audit. Even startpage has
- charcircuit 5y ago
  >you know when you use their products as intended you have no privacy This isn't true. Google's privacy policy is not lax as you suggest it is.
  - derevaunseraun 5y ago
    Could you clarify a bit? If you are logged in they store almost everything, as is evidenced by Google takeout.
    - charcircuit 5y ago
      If you give data to Google it's not like that data is now public.
    - cute_boi 5y ago
      Unfortunately, Google takeout doesn't seem to give all the data. Last time viber stored "secret data" in google drive. I tried to download everything via takeout. It doesn't seem to include those data.
- zagrebian 5y ago
marcodiego 5y ago
We need a p2p, independent, distributed, FLOSS search engine.
pojzon 5y ago
And here goes my reason to use DDG. Not because I pirate stuff, but because I hate censorship. Its only a matter of time till they implement the same filters as GGle. And in no way those filters stop dedicated ppl.
tandav 5y ago
Private search aggregation engine https://github.com/searx/searx https://github.com/searx/searx
- scarygliders 5y ago
  Exactly. Installed that recently and been using it for a few weeks - it's great to self-host one's own search aggregator and I'm very impressed with the results.
aunty_helen 5y ago
https://duckduckgo.com/traffic https://duckduckgo.com/traffic Have a look at a 50 day average and you can see their mistakes come to light. I've shifted off them recently. I used to be DDG for everything, then it became anything non-work related, then on phone with FF focus, now nothing. VPN and clearing cookies after browser close except for a few certain sites has replaced them.
ausbah 5y ago
scihub still shows up!
deleted 5y ago
[deleted]
zionic 5y ago
Just switched. Lets see how well brave search works out :)
- elforce002 5y ago
  I switched when they decided to "curate" their search.
- dirtyv 5y ago
  I absolutely do not trust Brave. I tried their browser on my iPhone for a few months. Always chose the most private settings, don't save history, etc, so whenever I opened the app there was no evidence it had even been used before. After one particular update, I started the app and it opened up literally dozens of tabs. Going through them I realized each tab was a site I had visited in the past. Left a nasty review on the App store that they never responded to, but they pushed out another update almost immediately. Never again. I wouldn't be the least bit surprised if years from now we find out the CIA was behind Brave, like they were with Crypto AG
  - syshum 5y ago
    Brave was started by one of the Founders of Mozilla.
    - einpoklum 5y ago
      With respect to the Mozilla project (not so much to its current state) - that does not mean much. Different people were involved with it, and that was 25 years ago.
      - syshum 5y ago
        Less than 10 years go... but ok It was also a time of Mozilla growth not decline like it is today Hell I would more charge Mozilla with being CIA front group than Brave based on their modern actions
    - bduerst 5y ago
deleted 5y ago
[deleted]
avipars 5y ago
I could be mistaken, but I believe that DDG sources data from Bing and other large search engines... If their sources, such as bing decide to remove a site, DDG gets affected by this as well.
swayvil 5y ago
Censoring my search results is the opposite of what I want in a search engine. I will not be using ddg anymore.
Chalbroth 5y ago
DDG sucks in many ways. Besides the engine performing quite poorly, it also relies on third parties and so will return filtered results they may not even control. They also never supported IPv6 and are hosted at Microsoft or Amazon. IMO, there is no credible search engine today.
- QuikAccount 5y ago
  > are hosted at Microsoft or Amazon I agree with most of what you are saying but this is silly. I get "Microsoft and Amazon bad" but basically saying "host your own infrastructure worldwide and index your own corpus or you are a bad actor is rather inane.
  - Chalbroth 5y ago
    They may not host their own infrastructure, even though I think something like DDG should. But they can choose a better network, better technically and ethically.
mfer 5y ago
They show up in the search results for me
jdright 5y ago
Ok, this is the start of the end for ddg, it was good while it lasted.
- mackrevinack 5y ago
  DuckDuckStop
deleted 5y ago
[deleted]
kovalevski 5y ago
so is ddg becoming a google ?
- ajyotirmay 5y ago
  Probably worse than Google at this point. At least Google doesn't hide behind "privacy respecting"
rejor121 5y ago
Well, not using DDg anymore, based on this and the comments down below. Any recommendations? Maybe I’ll give Lycos a try again xD
- swethmandava 5y ago
  Try https://you.com https://you.com - private and also feature rich
  - ajyotirmay 5y ago
    Same issue, youtube-dl.org doesn't show up in web results
    - richardsocher 5y ago
      Can you help me understand what you mean by it doesn't show up? It seems to be there like all over the place? In web, on the github app, in the reddit app results, etc. https://you.com/search?q=download%20youtube-dl https://you.com/search?q=download%20youtube-dl
NeonOverflow 5y ago
Well, now I'm definitely switching search engines.
vbezhenar 5y ago
I ditched DDG after it started to "downrank Russian disinformation". Thanks, I'll decide for myself where real disinformation belongs to. It became just another tool of western propaganda. I'm not sure if Google does that, but at least they don't paint themselves as good guys anymore.
- Marsymars 5y ago
  Russian disinformation is effectively spam. It’s akin to if your search results about copyright policy got polluted by SEO’d, backlink-filled, MPAA-funded blog spam about the evils of piracy and the Joy Of Disney that pushed down legitimate information about copyright policy.
  - iinnPP 5y ago
    The problem is that with only one side of a story, you never get a truthful story. The usefulness of having both sides of the story cannot be overstated. Otherwise the side telling the story doesn't even have to try and keep it realistic and can forgo any notion of truth. Allowing for more blatant lies to spread as truth. It is a very complex problem that I would say is best fought in early childhood development, by teaching our coming generations methods of discovering truth.
TheWill 5y ago
Quit using DDG when they decided to start censoring stuff. Been using brave search ever since, it's actually pretty good.
- bduerst 5y ago
  Didn't Brave get caught inserting their own affiliate links into search results when you clicked on them?
  - iinnPP 5y ago
    I would rank that pretty low on my list of things to worry about a search engine doing. Actually, to me, that would be relatively fair in terms of monetization of a search engine product. I would turn the feature on if asked by a reputable and well-meaning company. Especially in the case of a search engine seeing as they are providing the link afterall.
    - bduerst 5y ago
      Search engines rely on trust. Dark patterns that include shady things like exploitative URL insertions on clicking links are highly relevant.
funshed 5y ago
Have they? They whitelabled MSN/Bing search. It's probably Microsoft.
swethmandava 5y ago
https://you.com https://you.com - no ads, great privacy and let's YOU control your sources
- ikt 5y ago
  You have made 6 comments advertising you.com in a day? But for some reason you didn't bother to use your own search engine: https://you.com/search?q=youtube-dl&fromSearchBar=true https://you.com/search?q=youtube-dl&fromSearchBar=true https://yt-dl.org/ https://yt-dl.org/ is the homepage, and you.com has the same results as https://www.ecosia.org/search?q=youtube-dl&addon=firefox&addonversion=4.0.4 https://www.ecosia.org/search?q=youtube-dl&addon=firefox&add... and https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web It doesn't list the homepage...
  - richardsocher 5y ago
    Hey there, It looks like it's missing that url you mentioned but, I see tons of results on how to use it, where to download it, the correct github page with details, reddit results describing how to use it, etc.? Seems quite useful to me, no?
rowanG077 5y ago
Is there an alternative search engine that actually searches the web with removing search results?
- swethmandava 5y ago
  https://you.com https://you.com lets you like/dislike sources to see what you trust
  - ajyotirmay 5y ago
    Spam scam
lucb1e 5y ago
Core part of the article is at the bottom: > TorrentFreak reached out to DuckDuckGo to discover why these domain names are not showing up in its search results. At the time of publication, we have yet to hear back. It wouldn’t be a surprise if the move is copyright-related, though. So there isn't anything else known than "domains are delisted".
xnx 5y ago
I'm continually surprised by the amount of attention that HN gives DDG. They have no unique or interesting technology. The "privacy" claims are all self-attestation.
thanatos519 5y ago
I really appreciate these reports, because torrentfreak usually includes a list of the sites which were blocked, so I can add them to the list of torrent sites that I visit via unfiltered DNS and VPN proxy. I had forgotten about YTS until the recent BREIN (.nl) announcement, and I hadn't heard of Fmovies until today. Circumvention of these measures is so easy that I feel like it's making a haves/have-nots division on the Internet. Those of us with two or more clues are barely inconvenienced, and everyone else is blocked hard.
maydup-nem 5y ago
the fate of every successful venture, huh? getting screwed by political agenda and values TM.
- deleted 5y ago
  [deleted]
mrtweetyhack 5y ago
deleted 5y ago
[deleted]
jszymborski 5y ago
Well, after using DDG exclusively for web search since maybe 2014, I'll have to jump ship. I was never chuffed about them not using their own index, but removing priate sites and yt-dl, and the improving.duckduckgo.com business is exactly the type of behaviour I left Google for DDG over. I've liked what little of Brave Search I've used, but I'm not in love with Brave as a company either so perhaps I'll audition some of the paid web search options.
- ikt 5y ago
  It's interesting that Qwant never comes up in these discussions, they have their own engine and are privacy focused, looks very similar to Google but just never seems to be brought up?
  - jonashus 5y ago
    Same results on Qwant. No thepiratebay.org and not youtube dl's official site. The results seem very much identical.
stereoradonc 5y ago
Snake oil marketing? Its results have become less relevant to me now. I am using Kagi and Neeva for search results. Kagi is still in the beta but it does offer reasonably good output.
Barrera 5y ago
That's odd because even Google shows the homepage for YouTube-DL.
canyoudobetter 5y ago
DuckDuckGo has a cheat sheet specifically for youtube-dl so it seems quite unlikely they are actively trying to suppress content about youtube-dl: https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheatsheet&iax=cheatsheet https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat... There’s a lot of hate in this thread towards a company trying to do the right thing.
- account42 5y ago
  > https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat... "This Instant Answer was made by the DuckDuckHack Community." > There’s a lot of hate in this thread You would do yourself a favor if you tried to engage with criticism instead of deriding it as hate. If you think something is wrong you should be able to come up with convincing counter-arguments. > towards a company trying to do the right thing. A company that claims they are trying to do the right thing.
EchoReflection 5y ago
https://web.archive.org/web/20220313015243/https://www.washingtonpost.com/technology/2022/03/12/duckduckgo-russian-disinformation-unbiased-search-results/ https://web.archive.org/web/20220313015243/https://www.washi...
ddaalluu2 5y ago
Bye DDG, was nice knowing you
jmcnulty 5y ago
I've never understood how links can fall victim to copyright. A link is just a signpost instructing your browser where to go next. There is no copying involved, anywhere. Subjugating links with copyright law is morphing reality into fantasy.
thayne 5y ago
Is it actually DDG that is censoring them, or is it Bing and DDG is as well because it is using Bing results?
Brian_K_White 5y ago
What is an uncensored search engine then? If I search for something, and the search engine does not tell me about things which ARE there, then that is a defective search engine. It's fine to keep kids from getting ahold of any sharp objects but if I need a knife I need a knife and it's ridiculous for anyone else to decide to lie to me about the existence of knives. We are doing the Chinese firewall to ourselves just a bit slower.
Brian_K_White 5y ago
I don't see how this is excusable vs just adding an optional safety filter like for porn in the image search.
stuckkeys 5y ago
Stopped using them a while back. What is everyone using nowadays? Any good alternatives? I use a ra-pi with pihole for my dns. It blocks good amount of stuff
Sunspark 5y ago
This change must be region-limited, or just takes awhile to roll out. In Canada, I tried searching for 2 of the sites on DDG the story said were removed, and they both still show up.
darekkay 5y ago
Unfortunately, Bing (and thus DDG) removes proper websites from their index for unknown reasons, without any way to get this sorted out [0]. My blog (10 years, no spam content) has been delisted for 2 months now. The Bing webmaster tools don't display any error. After using DDG for 2-3 years, both as my main search engine and as OpenSearch for my blog, I had to switch back to Google. [0] https://techcommunity.microsoft.com/t5/microsoft-bing/help-why-my-site-has-been-removed-from-bing-s-search-results/m-p/2906441 https://techcommunity.microsoft.com/t5/microsoft-bing/help-w...
- ColinHayhurst 5y ago
  That's out-of-order on two counts. I see there are plenty of others in the same boat. Our thus far smaller but 5+ billion index has you covered: https://www.mojeek.com/search?q=site%3Awww.espositosoftware.it+software https://www.mojeek.com/search?q=site%3Awww.espositosoftware....
  - darekkay 5y ago
    Thanks, but that is not my site. Mine is darekkay.com and it isn't indexed by mojeek.
    - ColinHayhurst 5y ago
      OK, thanks. Why did you think that? Looks like, at least some, is indexed: https://www.mojeek.com/search?q=darekkay+site%3Adarekkay.com https://www.mojeek.com/search?q=darekkay+site%3Adarekkay.com
      - darekkay 5y ago
        I've searched just for site:domain (without any specific keyword), which both Google and Bing support.
        ColinHayhurst 5y ago
        Thanks for pointing that out. We'll add support for that too.
Shadonototra 5y ago
Proprietary software Refused every independent audits Political censorship -- They are now moving to that other project, so predictable.. intelligence really is dead in the US
jpswade 5y ago
Doesn’t DDG just use Bing under the hood?
02020202 5y ago
ramyon456 5y ago
they've also started completely censoring nubile/18 (aka legal) porn images since a couple weeks
ComradePhil 5y ago
They explicitly said they down rank "disinformation". Take it how you want: https://mobile.twitter.com/yegg/status/1501716484761997318 https://mobile.twitter.com/yegg/status/1501716484761997318
HNHatesUsers 5y ago
thoweuoi23324 5y ago
Quite ironic that the only 'free' search engine left is Yandex. So much for freedom and democracy (TM).
- account42 5y ago
  I hope you are not implying that Yandex does not censor its search results.
cobbaut 5y ago
Really? Just searched "pirate bay" on DDG. First result is thepiratebay.org.
- busyboy 5y ago
  Seems like searching "fmovies" also gives some results.
sylware 5y ago
It seems duckduckgo is a microsoft proxy: as far as I know, its advertising agency is the one from microsoft.
xtat 5y ago
RIP DDG
T3RMINATED 5y ago
Trouble_007 5y ago
https://news.ycombinator.com/item?id=31059882 https://news.ycombinator.com/item?id=31059882 : DDG: We are not “purging” any media outlets from results >Gabriel Weinberg @yegg Apr 17, 2022 10:18AM UTC Hoping to clear up some misconceptions about our private search engine. ... < Gabriel Weinberg, @yegg CEO & Founder @DuckDuckGo - Nitter : https://nitter.net/yegg/ https://nitter.net/yegg/
- itsTyrion 5y ago
  The mentioned sites aren't filtered from DDG for me
itsTyrion 5y ago
I tested every mentioned site, nothing is filtered in DDG for me. Hoax or just based on country?

duckduckgone

einpoklum 5y ago
Damn it! You beat me to this quip! Here's your damn +1. Grrr...

I don’t condone the actions of the Russian government; however: If you want better results from pirate sites, try Yandex.com. Quite good for finding torrents.

Jerry2 5y ago
>Quite good for finding torrents. That's because there's no DMCA law there and they're not taking down thousands upon thousands of pages like Google is forced to do every day. [1] [1] https://www.lumendatabase.org/ https://www.lumendatabase.org/
- codedokode 5y ago
  Yandex has an agreement with copyright holders and removes sites with pirated content from results. But those are Russian copyright holders and they mostly care about content in Russian. Also Yandex removes sites that are banned in Russia, for example BBC in Russian or Navalny's site.
probably_wrong 5y ago
I used to be able to find MP3s with Google, until one day it simply stopped working. Then I moved to Baidu, which worked for a while. I now use DDG to find movies, and yet looks like the curtain may be closing on that one too. I fear it might only be a matter of time before Yandex stops working too.
- Markoff 5y ago
  there are telegram bots for it though, personally I don't use them but people swear on them btw Baidu is extremely slow due to GFW
  - itsTyrion 5y ago
    true that. I tried to look something up on Baidu once (from Germany) and gave up after it loaded for a full minute
draugadrotten 5y ago
Russian government has always appreciated https://en.wikipedia.org/wiki/Kompromat https://en.wikipedia.org/wiki/Kompromat so be careful what you search for.
seanw444 5y ago
I love how we have to preface our approval of something foreign with "now I'm not a Nazi, but...'
- cpach 5y ago
  The obvious reason I used that disclaimer was the fact that Russia’s government decided to invade Ukraine.
Markoff 5y ago
why would you look for torrents in search engine? you will anyway find most of them dead you are much better off with Knaben on EU domain, same results can be achieved with add-ons in qbit, then there are special software not worth my time (Jackett) but I agree I find myself using yandex much more nowadays as alternative
- cpach 5y ago
  Yandex made it possible for me to find extremely rare movies that are not to be found elsewhere. So I have found it very useful.

mdaniel 5y ago

As for the YouTube-DL part specifically, I'm pretty sure that's a side-effect of them being dependent upon Bing for the actual index: https://www.bing.com/search?q=site%3Ayoutube-dl.org https://www.bing.com/search?q=site%3Ayoutube-dl.org Also, _come on_; if someone gives up after one search term, which also includes advanced site-restricting syntax, there's no way they'd be able to operate youtube-dl anyway: https://duckduckgo.com/?q=youtube-dl&ia=web https://duckduckgo.com/?q=youtube-dl&ia=web

muhammadusman 5y ago
Yeah idk why someone would type in the full URL of a site and not just the term they're looking for in a search engine...what a dumb way to test that on their part.
- miloignis 5y ago
  The point isn't that you can't reach the website by typing in the search term, the point is that they've deindexed the site. If you search for youtube-dl, neither youtube-dl.org nor yt-dl.org come up in the results, as far as I can see. You can still access it from the results via other sites, like github and wikipedia (which even makes the real website pop up in the info square, funnily enough), but the search results themselves do not contain any links to the main website.
- daxuak 5y ago
  What if I want to search what sites have mentioned those urls?
  - lolinder 5y ago
    That's still the wrong syntax.
- ykonstant 5y ago
  Absolutely! Why on earth would you test a dumb way of doing things on a user-facing service?! slams hand on desk Irrelevant!

deleted 5y ago

[deleted]

ravenstine 5y ago

I'm just waiting for the day they announce an NFT or a "trusted partners" program with establishment media entities. Come on, DDG, you've come this far, so truly jump the shark for our amusement.

bouncycastle 5y ago
Which would be ironic since Nfts cannot be censored (they exist on a decentralized censorship resistant database)

mikece 5y ago

I thought yt-dlp was recommended over youtube-dl these days. Any chance DDG suppressing a less-preferred program?

ajyotirmay 5y ago
It's a search engine, not a popularity ranking system.

Vladimof 5y ago

thematrixturtle 5y ago
Google pushed the envelope on how silly the name for a major company can be, but DuckDuckGo is unlikely to ever hit mass market success with that name. (Or, more importantly, the current implementation.)
- Vladimof 5y ago
  Google even gave duck.com to them, for free, I think I used duckduckgo on and off for probably a year ... might as well use Bing if you like it
  - iinnPP 5y ago
    I don't think Google does anything for free. Sure provides a good reason not to use it.
- gigglesupstairs 5y ago
  It also owns www.duck.com. Not sure why they’re still sticking with DuckDuckGo.
NoImmatureAdHom 5y ago
tbh I find it to be better than Google
- Vladimof 5y ago
  tbh I find that all search engines are worst then Google was 10 years ago (including Google)
  - NoImmatureAdHom 5y ago
    Agreed. Gimme goddamn booleans that work!

nmilo 5y ago

Youtube-dl's site is yt-dl.org, not youtube-dl.org. Not surprised the second one doesn't show up. Not sure why this blogspam is here on HN either.

miloignis 5y ago
yt-dl.org also seems not to appear: https://duckduckgo.com/?q=site%3Ayt-dl.org&t=h_&ia=web https://duckduckgo.com/?q=site%3Ayt-dl.org&t=h_&ia=web
mdaniel 5y ago
They are both bing suppressed, and as best I can tell are synonymous $ dig +short youtube-dl.org. 95.143.172.170 $ dig +short yt-dl.org. 95.143.172.170 as for the blogspam, I'm sure everyone loves a good free speech suppression story, but yeah, I wish HN offered downvotes on submissions, too
syoc 5y ago
I agree that was not a high quality post from TorrenFreak. I do however also find the blogspam label to be unfair. TF has pushed some important agendas over the years.

slig 5y ago

So much for a Google alternative.

Apreche 5y ago

Someone should make a search engine that only indexes sites that Google and DuckDuckgo do not index. It would serve sort of the same purpose as like, lists of banned books.

bufferoverflow 5y ago
It will be a lot of spam with a bit of useful sites sprinkled randomly.
jfengel 5y ago
You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links? Yeah, I guess that would probably be useful for something. Academic research into just how awful people are, perhaps. And, of course, as a blackhole list. Just make sure you get a bulk discount on disk space. Such a list would be huge.
- theknocker 5y ago
- darkteflon 5y ago
  To be fair, “SEO-driven gibberish with back-links” is an apt description of Google’s actual results these days.
- Brian_K_White 5y ago
  "You want a list of spam sites, phishing, malware, and SEO-driven gibberish with back-links?" Yes, exactly, I want exactly whatever I search for, including that if that's what I search for. If I search for something which exists, then I want the knowledge of that existence. The value judgement is no one else's business unless I search for "X but only where approved by any governments or companies with any political or financial interest in controlling what I know." When it turns out that I would never choose to search for that, that is the proof that it's wrong to be doing it.
  - 5y ago

twoxproblematic 5y ago

eimrine 5y ago

Who really needs a search engine nowadays? The google or ddg or whatever is just a search aggregator for making no need to open Wikipedia for searching who is Franko then open weather.com for figuring out the weather next weekend then open SO for quitting vim and so on. I can not remember when I really searched something general last time which is impossible to find via single-site search. And of course free porn videos and other pirated stuff and cool-hacker-utils is never been searched using government-friendly search businesses.

jjulius 5y ago
It's cool that you've had enough experience to just know where everything that you need is. Not everybody has the same experience with finding things on the internet as you, nor is the average internet user anywhere near as technically-minded as you.
drexlspivey 5y ago
> Who really needs a search engine nowadays? Apparently a lot of people since google processes 6B searches every day?
- thematrixturtle 5y ago
  I have no idea what the actual number is, but that sounds low. The average HNer probably hits three digits every day easily.
- eimrine 5y ago
  6B showings of ad-loaded webpage does not mean 6B successful results shown. And my claim is that most of successful google searches is just a wrong choice of a search engine.
BlackestNight 5y ago

kingcharles 5y ago

What about archive.org? Will they delist that too? That definitely contains a lot of copyright-infringing data. Where will this end? Pirate Bay actually contains a lot of non-infringing items too. Who watches the watchers?

yjftsjthsd-h 5y ago
> Pirate Bay actually contains a lot of non-infringing items too. Wait, what have they got? Actual Linux ISOs?
- robbedpeter 5y ago
  And public domain content, and content licensed for free distribution, and abandoned software. The good stuff is there too, but the legit free stuff isn't all bad.
KSPAtlas 5y ago
You can literally get Windows keys on archive.org, not that I use Windows anyways
- ajyotirmay 5y ago
  10yo me would have loved this information. Linux FTW

walrus01 5y ago

isn't duckduckgo just a skin on top of bing?

HigherPlain 5y ago
It's Bing.
bduerst 5y ago
They claim to have results from over "400 sources" but then don't list them.

anm89 5y ago

And I have removed duck duck go as my default browser

anm89 5y ago
*search engine
- ajyotirmay 5y ago
  What are you switching to?

thematrixturtle 5y ago

I suspect this is largely due to DDG using Bing under the hood, which has led to similar weirdness in the past, eg all major porn sites disappearing from the results in Singapore (while Google still showed them, mind you!). Still super disappointing though, and yet another reason why trying to build a better search engine on top of someone else's tech is a non-starter.

KMag 5y ago
Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it on over the visible web. You need to bootstrap somehow form an existing index if you want to get anywhere in any reasonable time. If there's a market for alternative search engines out there, AWS is already serving so much of the web that Amazon should really provide crawling and basic indexing (for a fee, of course) so it's done once for everyone, with Amazon Lambda processing to allow search engines to customize their indexing. I'm not sure if it would make them much money, but it makes more sense to start up a search engine on AWS using Amazon's crawl and basic indexing vs. using a search engine competitor's crawl and indexing.
- post-it 5y ago
  That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO. Awful for competition and whatnot.
  - KMag 5y ago
    Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition. As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.

barnabee 5y ago

I don’t need these results but I don’t want to support this behaviour/trend. Can anyone recommend a decent non-Google alternative.

ColinHayhurst 5y ago
Your choices are limited, and very few independent indexes. Most options are Bing alternatives: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
- mdaniel 5y ago
  That was fascinating, thanks so much for sharing it
theandrewbailey 5y ago
https://search.brave.com/ https://search.brave.com/
nabeards 5y ago
https://presearch.org https://presearch.org
- xigoi 5y ago
  Cryptocurrency? No, thanks.
lolinder 5y ago
I've been using Kagi (https://kagi.com https://kagi.com) for a few months and it's been great. It's free while in beta but will be funded by subscription fees afterward. The killer feature for me is the ability to up-rank and down-rank sites in my personal results, which has been really helpful for curating my tech-related searches. It allows me to quickly find high-quality docs and eliminate the garbage.
- 5y ago

babypuncher 5y ago

What is with this blogspam? YouTube-DL comes up just fine in search results (https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web). So does Pirate Bay (https://duckduckgo.com/?q=pirate+bay&t=h_&ia=web https://duckduckgo.com/?q=pirate+bay&t=h_&ia=web). The headline is outright false.

Flollop 5y ago
the complaint in the article is around the serach engine's "search within site" feature. Compare the following two: - https://duckduckgo.com/?q=site%3Athepiratebay.org&t=h_&ia=web https://duckduckgo.com/?q=site%3Athepiratebay.org&t=h_&ia=we... - https://duckduckgo.com/?q=site%3Aycombinator.com&t=h_&ia=web https://duckduckgo.com/?q=site%3Aycombinator.com&t=h_&ia=web
miloignis 5y ago
The YouTube-DL website notably does not come up in those search results, it's their GitHub page that does. The website does appear in the sidebar, because it pulls the sidebar data from Wikipedia, and YouTube-DL's Wikipedia page does come up. Note that if you do a google search (https://www.google.com/search?q=youtube-dl https://www.google.com/search?q=youtube-dl) their website is the first result.
- gigglesupstairs 5y ago
  Both the website and the GitHub are top two results at my end, in India.
  - miloignis 5y ago
    Interesting, mine are github, videohelp.com, and then ottverse. I imagine that adds credence to the theory that it's because of Bing's index removing them - maybe they use a different index in India?
  - omgmajk 5y ago
    Their main website is not coming up here (sweden). Only the github and some random fringe-sites, obvious scams and readthedocs.

rolph 5y ago

a search using; anyterm :: torrent still returns alot of results for me sofar. and so does youtube-dl; top result is : github.com/ytdl-org

ajyotirmay 5y ago
That's the github link, not youtube-dl website :)
- rolph 5y ago
  I can still reach it yt-dl.org and as youve noticed the repo is still part of search results as well.

srvmshr 5y ago

Serious Question: How is the DDG search structured? Is it a cosmetic skin over Bing, or is it aggregating from other sites like Yahoo, ecosia etc additionally? If it is just Bing under the hood, how does it exist as business entity. I am sure MS will take some action to consolidate their search share rather than seeing splintered.

marpstar 5y ago
They aggregate from "over 400 sources" and run their own crawler: https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
- bduerst 5y ago
  Do they list their "sources"? It reads like they are saying a website = source, and the hyperlink for the sources is dead.
- cato_the_elder 5y ago
  That's just their corporate propaganda. For the most part, it's Bing with some additional features. Their crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [1] [1]: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html#fn:2 https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
  - etataetaet 5y ago
    To be fair the instant answers are usually quite good!
cato_the_elder 5y ago
> If it is just Bing under the hood, how does it exist as business entity. They provide an alternative branding, targeted at "privacy-aware" users and hipsters.
HigherPlain 5y ago

deleted 5y ago

[deleted]

AdmiralAsshat 5y ago

I've started hearing ads for DuckDuckGo on NPR, of all places. Could be completely coincidental, but I find the timing suspicious. Do you think they're gearing up for an IPO?

derekbaker783 5y ago
I think they've advertised there for a while.
ronsor 5y ago
I've seen DuckDuckGo ads on network TV.

zekrioca 5y ago

At least when querying "youtube-dl" it shows a small snippet to the right where one can see the URL..

Kiro 5y ago

Funny how the narrative on DDG has changed. I used to get downvoted to oblivion for merely mentioning they used Bing under the hood. Where are all the people that used to defend them so vigorously? I wouldn't be surprised if it's the same people now hating on them.

tibyat 5y ago
qiskit 5y ago
> I wouldn't be surprised if it's the same people now hating on them. It seems like those people are still here defending ddg. They are even attacking the blog. Weird. I use ddg as my default search and hope they are "privacy-first" as they claim ( thought I have my doubts ). At this point what other option is there? DDG search is noticably censored since they rely on bing. It's obvious to anyone who uses ddg regularly. Not sure why people here are making excuses for ddg or pretending otherwise. What I would give to get the google of old. It's sad looking back on what google used to be. All the freedom and optimism of the 2000s is definitely gone.
- seanw444 5y ago
  Searx. Use Searx. Does your search engine of choice fail to get results that the others pick up? Searx grabs all of them and compiles them together. Don't miss another search result! Also, proxying increases privacy. Also, it's open source and self-hostable.
  - prmoustache 5y ago
    Being open source doesn't mean the people running the public searx instances aren't tracking you. Self-hosting makes you lose all the benefits of proxying. In the end be it ddg or searx you can only rely on some third-party trust.
    - seanw444 5y ago

autoexec 5y ago

It's amazing how much collateral damage is caused by our horrible copyright laws. Mostly just so the MPA/RIAA can protect their roles as gatekeepers of what we're allowed to see and hear. They can put enormous pressure on even the wealthiest and most powerful companies to act as copyright police on their behalf. Even Google is afraid of them. ISPs are forced to spend huge amounts of time and money working for them. Now duckduckgo is being strong armed into doing a bunch of free work for them too? Maintaining lists of websites and domains to block and removing links to even non-infringing material like youtube-dl just to keep from being sued into the ground. I don't know what it'll take to rein in these guys, but I doubt the courts will be the ones to do it. So far courts seem fine with the idea that ISPs must permanently ban users from their service over nothing but repeated unsubstantiated claims of infringements which is an insane amount of power to give any industry. Has any US politician ever run on a platform that includes copyright reform?

rglullis 5y ago
Lawrence Lessig
- autoexec 5y ago
  I forgot about that! What a missed opportunity that was for this country. He's exactly the sort of person we need to see more of in politics.
  - perardi 5y ago
    His campaign was much more about electoral reform than copyright reform. https://en.wikipedia.org/wiki/Lawrence_Lessig_2016_presidential_campaign https://en.wikipedia.org/wiki/Lawrence_Lessig_2016_president... Which is broadly more important than copyright reform…and broadly Americans don’t seem to particularly care about either of them all that much.
    - wussboy 5y ago
      Either would have been a win as far as I’m concerned.

teekert 5y ago

Hmm this will affect me, I always type tpb in ddg to go there. I’m also happy my provider is small and freedom loving so it isn’t blocked (neither is RT for that matter).

JohnWhigham 5y ago

What is going on with this company in the past several months? Completely destroying any goodwill they had

zodzedzi 5y ago

I use DDG as my default search engine, along with NoScript in the browser. Often when I visit a new website, I peruse the (long) list of domains that the site is trying to pull javascripts from. I keep most of those source sites in UNTRUSTED status (including some of the big names in search/ads/etc). But I've always had DDG in the TRUSTED category because I had only seen its javascript before on the main DDG website. (Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com"). But recently I started noticing some websites pulling javascript from DDG (I don't remember which sites). So now I was wondering if DDG is getting into the tracking business, since they're now having their javascripts load from third party sites. Obviously this is anecdotal. But does anyone know if they are indeed beginning to track?

stjohnswarts 5y ago
Can you give us a list (or partial list) of sites that are pulling scripts from duckduckgo? We can look at what they're trying to do.
- zodzedzi 5y ago
  I don't remember the sites; I'll try to find them again, and will share here if I do. I remember seeing 3 sites within an hour, and deciding to change the DDG setting to TEMP:TRUSTED afterwards.
  - autoexec 5y ago
    The nice thing about DDG is that if you're willing to sacrifice some functionality it can still be used without JS at all (which is how I use it)
z3c0 5y ago
> Unfortunately NoScript has a limitation that you can't tell it to "only TRUST javascript from example.com when I'm visiting example.com" I was under the impression that the custom option allowed this. Am I misunderstanding the point of this option?

derevaunseraun 5y ago

Question: does anyone actually know what DDG does with user data? Like they market themselves as a "privacy respecting" search engine, but how much of this is truth? I'd imagine there's good money in convincing people they have privacy because then they'll provide more interesting data. Has the company ever been audited? Why should they be trusted to not compromise user privacy? Imo at least Google is honest: you know when you use their products as intended you have no privacy, and they don't try to hide this Edit: since DDG isn't open source like searx, how do we know there is ANY truth to their marketing claims? Edit: Just for accuracy, the browser extensions are open source. But as far as I know, the actual search engine isn't Edit: They made over 100 million in 2020. They clearly can (and should) get an independent audit. It's shocking that they haven't had a single audit. Even startpage has

charcircuit 5y ago
>you know when you use their products as intended you have no privacy This isn't true. Google's privacy policy is not lax as you suggest it is.
- derevaunseraun 5y ago
  Could you clarify a bit? If you are logged in they store almost everything, as is evidenced by Google takeout.
  - charcircuit 5y ago
    If you give data to Google it's not like that data is now public.
  - cute_boi 5y ago
    Unfortunately, Google takeout doesn't seem to give all the data. Last time viber stored "secret data" in google drive. I tried to download everything via takeout. It doesn't seem to include those data.
zagrebian 5y ago

marcodiego 5y ago

We need a p2p, independent, distributed, FLOSS search engine.

pojzon 5y ago

And here goes my reason to use DDG. Not because I pirate stuff, but because I hate censorship. Its only a matter of time till they implement the same filters as GGle. And in no way those filters stop dedicated ppl.

tandav 5y ago

Private search aggregation engine https://github.com/searx/searx https://github.com/searx/searx

scarygliders 5y ago
Exactly. Installed that recently and been using it for a few weeks - it's great to self-host one's own search aggregator and I'm very impressed with the results.

aunty_helen 5y ago

https://duckduckgo.com/traffic https://duckduckgo.com/traffic Have a look at a 50 day average and you can see their mistakes come to light. I've shifted off them recently. I used to be DDG for everything, then it became anything non-work related, then on phone with FF focus, now nothing. VPN and clearing cookies after browser close except for a few certain sites has replaced them.

ausbah 5y ago

scihub still shows up!

deleted 5y ago

[deleted]

zionic 5y ago

Just switched. Lets see how well brave search works out :)

elforce002 5y ago
I switched when they decided to "curate" their search.
dirtyv 5y ago
I absolutely do not trust Brave. I tried their browser on my iPhone for a few months. Always chose the most private settings, don't save history, etc, so whenever I opened the app there was no evidence it had even been used before. After one particular update, I started the app and it opened up literally dozens of tabs. Going through them I realized each tab was a site I had visited in the past. Left a nasty review on the App store that they never responded to, but they pushed out another update almost immediately. Never again. I wouldn't be the least bit surprised if years from now we find out the CIA was behind Brave, like they were with Crypto AG
- syshum 5y ago
  Brave was started by one of the Founders of Mozilla.
  - einpoklum 5y ago
    With respect to the Mozilla project (not so much to its current state) - that does not mean much. Different people were involved with it, and that was 25 years ago.
    - syshum 5y ago
      Less than 10 years go... but ok It was also a time of Mozilla growth not decline like it is today Hell I would more charge Mozilla with being CIA front group than Brave based on their modern actions
  - bduerst 5y ago

deleted 5y ago

[deleted]

avipars 5y ago

I could be mistaken, but I believe that DDG sources data from Bing and other large search engines... If their sources, such as bing decide to remove a site, DDG gets affected by this as well.

swayvil 5y ago

Censoring my search results is the opposite of what I want in a search engine. I will not be using ddg anymore.

Chalbroth 5y ago

DDG sucks in many ways. Besides the engine performing quite poorly, it also relies on third parties and so will return filtered results they may not even control. They also never supported IPv6 and are hosted at Microsoft or Amazon. IMO, there is no credible search engine today.

QuikAccount 5y ago
> are hosted at Microsoft or Amazon I agree with most of what you are saying but this is silly. I get "Microsoft and Amazon bad" but basically saying "host your own infrastructure worldwide and index your own corpus or you are a bad actor is rather inane.
- Chalbroth 5y ago
  They may not host their own infrastructure, even though I think something like DDG should. But they can choose a better network, better technically and ethically.

mfer 5y ago

They show up in the search results for me

jdright 5y ago

Ok, this is the start of the end for ddg, it was good while it lasted.

mackrevinack 5y ago
DuckDuckStop

deleted 5y ago

[deleted]

kovalevski 5y ago

so is ddg becoming a google ?

ajyotirmay 5y ago
Probably worse than Google at this point. At least Google doesn't hide behind "privacy respecting"

rejor121 5y ago

Well, not using DDg anymore, based on this and the comments down below. Any recommendations? Maybe I’ll give Lycos a try again xD

swethmandava 5y ago
Try https://you.com https://you.com - private and also feature rich
- ajyotirmay 5y ago
  Same issue, youtube-dl.org doesn't show up in web results
  - richardsocher 5y ago
    Can you help me understand what you mean by it doesn't show up? It seems to be there like all over the place? In web, on the github app, in the reddit app results, etc. https://you.com/search?q=download%20youtube-dl https://you.com/search?q=download%20youtube-dl

NeonOverflow 5y ago

Well, now I'm definitely switching search engines.

vbezhenar 5y ago

I ditched DDG after it started to "downrank Russian disinformation". Thanks, I'll decide for myself where real disinformation belongs to. It became just another tool of western propaganda. I'm not sure if Google does that, but at least they don't paint themselves as good guys anymore.

Marsymars 5y ago
Russian disinformation is effectively spam. It’s akin to if your search results about copyright policy got polluted by SEO’d, backlink-filled, MPAA-funded blog spam about the evils of piracy and the Joy Of Disney that pushed down legitimate information about copyright policy.
- iinnPP 5y ago
  The problem is that with only one side of a story, you never get a truthful story. The usefulness of having both sides of the story cannot be overstated. Otherwise the side telling the story doesn't even have to try and keep it realistic and can forgo any notion of truth. Allowing for more blatant lies to spread as truth. It is a very complex problem that I would say is best fought in early childhood development, by teaching our coming generations methods of discovering truth.

TheWill 5y ago

Quit using DDG when they decided to start censoring stuff. Been using brave search ever since, it's actually pretty good.

bduerst 5y ago
Didn't Brave get caught inserting their own affiliate links into search results when you clicked on them?
- iinnPP 5y ago
  I would rank that pretty low on my list of things to worry about a search engine doing. Actually, to me, that would be relatively fair in terms of monetization of a search engine product. I would turn the feature on if asked by a reputable and well-meaning company. Especially in the case of a search engine seeing as they are providing the link afterall.
  - bduerst 5y ago
    Search engines rely on trust. Dark patterns that include shady things like exploitative URL insertions on clicking links are highly relevant.

funshed 5y ago

Have they? They whitelabled MSN/Bing search. It's probably Microsoft.

swethmandava 5y ago

https://you.com https://you.com - no ads, great privacy and let's YOU control your sources

ikt 5y ago
You have made 6 comments advertising you.com in a day? But for some reason you didn't bother to use your own search engine: https://you.com/search?q=youtube-dl&fromSearchBar=true https://you.com/search?q=youtube-dl&fromSearchBar=true https://yt-dl.org/ https://yt-dl.org/ is the homepage, and you.com has the same results as https://www.ecosia.org/search?q=youtube-dl&addon=firefox&addonversion=4.0.4 https://www.ecosia.org/search?q=youtube-dl&addon=firefox&add... and https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web https://duckduckgo.com/?q=youtube-dl&t=h_&ia=web It doesn't list the homepage...
- richardsocher 5y ago
  Hey there, It looks like it's missing that url you mentioned but, I see tons of results on how to use it, where to download it, the correct github page with details, reddit results describing how to use it, etc.? Seems quite useful to me, no?

rowanG077 5y ago

Is there an alternative search engine that actually searches the web with removing search results?

swethmandava 5y ago
https://you.com https://you.com lets you like/dislike sources to see what you trust
- ajyotirmay 5y ago
  Spam scam

lucb1e 5y ago

Core part of the article is at the bottom: > TorrentFreak reached out to DuckDuckGo to discover why these domain names are not showing up in its search results. At the time of publication, we have yet to hear back. It wouldn’t be a surprise if the move is copyright-related, though. So there isn't anything else known than "domains are delisted".

xnx 5y ago

I'm continually surprised by the amount of attention that HN gives DDG. They have no unique or interesting technology. The "privacy" claims are all self-attestation.

thanatos519 5y ago

I really appreciate these reports, because torrentfreak usually includes a list of the sites which were blocked, so I can add them to the list of torrent sites that I visit via unfiltered DNS and VPN proxy. I had forgotten about YTS until the recent BREIN (.nl) announcement, and I hadn't heard of Fmovies until today. Circumvention of these measures is so easy that I feel like it's making a haves/have-nots division on the Internet. Those of us with two or more clues are barely inconvenienced, and everyone else is blocked hard.

maydup-nem 5y ago

the fate of every successful venture, huh? getting screwed by political agenda and values TM.

deleted 5y ago
[deleted]

mrtweetyhack 5y ago

deleted 5y ago

[deleted]

jszymborski 5y ago

Well, after using DDG exclusively for web search since maybe 2014, I'll have to jump ship. I was never chuffed about them not using their own index, but removing priate sites and yt-dl, and the improving.duckduckgo.com business is exactly the type of behaviour I left Google for DDG over. I've liked what little of Brave Search I've used, but I'm not in love with Brave as a company either so perhaps I'll audition some of the paid web search options.

ikt 5y ago
It's interesting that Qwant never comes up in these discussions, they have their own engine and are privacy focused, looks very similar to Google but just never seems to be brought up?
- jonashus 5y ago
  Same results on Qwant. No thepiratebay.org and not youtube dl's official site. The results seem very much identical.

stereoradonc 5y ago

Snake oil marketing? Its results have become less relevant to me now. I am using Kagi and Neeva for search results. Kagi is still in the beta but it does offer reasonably good output.

Barrera 5y ago

That's odd because even Google shows the homepage for YouTube-DL.

canyoudobetter 5y ago

DuckDuckGo has a cheat sheet specifically for youtube-dl so it seems quite unlikely they are actively trying to suppress content about youtube-dl: https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheatsheet&iax=cheatsheet https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat... There’s a lot of hate in this thread towards a company trying to do the right thing.

account42 5y ago
> https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat https://duckduckgo.com/?q=youtubedl%20cheat%20sheet&ia=cheat... "This Instant Answer was made by the DuckDuckHack Community." > There’s a lot of hate in this thread You would do yourself a favor if you tried to engage with criticism instead of deriding it as hate. If you think something is wrong you should be able to come up with convincing counter-arguments. > towards a company trying to do the right thing. A company that claims they are trying to do the right thing.

EchoReflection 5y ago

https://web.archive.org/web/20220313015243/https://www.washingtonpost.com/technology/2022/03/12/duckduckgo-russian-disinformation-unbiased-search-results/ https://web.archive.org/web/20220313015243/https://www.washi...

ddaalluu2 5y ago

Bye DDG, was nice knowing you

jmcnulty 5y ago

I've never understood how links can fall victim to copyright. A link is just a signpost instructing your browser where to go next. There is no copying involved, anywhere. Subjugating links with copyright law is morphing reality into fantasy.

thayne 5y ago

Is it actually DDG that is censoring them, or is it Bing and DDG is as well because it is using Bing results?

Brian_K_White 5y ago

What is an uncensored search engine then? If I search for something, and the search engine does not tell me about things which ARE there, then that is a defective search engine. It's fine to keep kids from getting ahold of any sharp objects but if I need a knife I need a knife and it's ridiculous for anyone else to decide to lie to me about the existence of knives. We are doing the Chinese firewall to ourselves just a bit slower.

Brian_K_White 5y ago

I don't see how this is excusable vs just adding an optional safety filter like for porn in the image search.

stuckkeys 5y ago

Stopped using them a while back. What is everyone using nowadays? Any good alternatives? I use a ra-pi with pihole for my dns. It blocks good amount of stuff

Sunspark 5y ago

This change must be region-limited, or just takes awhile to roll out. In Canada, I tried searching for 2 of the sites on DDG the story said were removed, and they both still show up.

darekkay 5y ago

Unfortunately, Bing (and thus DDG) removes proper websites from their index for unknown reasons, without any way to get this sorted out [0]. My blog (10 years, no spam content) has been delisted for 2 months now. The Bing webmaster tools don't display any error. After using DDG for 2-3 years, both as my main search engine and as OpenSearch for my blog, I had to switch back to Google. [0] https://techcommunity.microsoft.com/t5/microsoft-bing/help-why-my-site-has-been-removed-from-bing-s-search-results/m-p/2906441 https://techcommunity.microsoft.com/t5/microsoft-bing/help-w...

ColinHayhurst 5y ago
That's out-of-order on two counts. I see there are plenty of others in the same boat. Our thus far smaller but 5+ billion index has you covered: https://www.mojeek.com/search?q=site%3Awww.espositosoftware.it+software https://www.mojeek.com/search?q=site%3Awww.espositosoftware....
- darekkay 5y ago
  Thanks, but that is not my site. Mine is darekkay.com and it isn't indexed by mojeek.
  - ColinHayhurst 5y ago
    OK, thanks. Why did you think that? Looks like, at least some, is indexed: https://www.mojeek.com/search?q=darekkay+site%3Adarekkay.com https://www.mojeek.com/search?q=darekkay+site%3Adarekkay.com
    - darekkay 5y ago
      I've searched just for site:domain (without any specific keyword), which both Google and Bing support.
      - ColinHayhurst 5y ago
        Thanks for pointing that out. We'll add support for that too.

Shadonototra 5y ago

Proprietary software Refused every independent audits Political censorship -- They are now moving to that other project, so predictable.. intelligence really is dead in the US

jpswade 5y ago

Doesn’t DDG just use Bing under the hood?

02020202 5y ago

ramyon456 5y ago

they've also started completely censoring nubile/18 (aka legal) porn images since a couple weeks

ComradePhil 5y ago

They explicitly said they down rank "disinformation". Take it how you want: https://mobile.twitter.com/yegg/status/1501716484761997318 https://mobile.twitter.com/yegg/status/1501716484761997318

HNHatesUsers 5y ago

thoweuoi23324 5y ago

Quite ironic that the only 'free' search engine left is Yandex. So much for freedom and democracy (TM).

account42 5y ago
I hope you are not implying that Yandex does not censor its search results.

cobbaut 5y ago

Really? Just searched "pirate bay" on DDG. First result is thepiratebay.org.

busyboy 5y ago
Seems like searching "fmovies" also gives some results.

sylware 5y ago

It seems duckduckgo is a microsoft proxy: as far as I know, its advertising agency is the one from microsoft.

xtat 5y ago

RIP DDG

T3RMINATED 5y ago

Trouble_007 5y ago

https://news.ycombinator.com/item?id=31059882 https://news.ycombinator.com/item?id=31059882 : DDG: We are not “purging” any media outlets from results >Gabriel Weinberg @yegg Apr 17, 2022 10:18AM UTC Hoping to clear up some misconceptions about our private search engine. ... < Gabriel Weinberg, @yegg CEO & Founder @DuckDuckGo - Nitter : https://nitter.net/yegg/ https://nitter.net/yegg/

itsTyrion 5y ago
The mentioned sites aren't filtered from DDG for me

itsTyrion 5y ago

I tested every mentioned site, nothing is filtered in DDG for me. Hoax or just based on country?