22 ms·

Techdirt has been deleted from Bing and DuckDuckGo [fixed]

220 points by lehi 3y ago

HWR_14 3y ago
I've noticed DDG dropping a lot of sites over the past month or so.
- yegg 3y ago
  Do you know what they are? Happy to investigate along with TechDirt and that might help.
  - HWR_14 3y ago
    I don't recall at the moment. If I remember I'll let you know.
    - mdaniel 3y ago
      a more sustainable approach would be to use the "Share Feedback" in the (regrettably positioned) lower right corner of the search results page. The story I've heard is that actual humans look at that feedback
      - yegg 3y ago
        Yes, that's right -- we do!
VHRanger 3y ago
As a long time DDG user, I was looking for a reason to try Kagi out, and this nonsense is finally pushing me over the fence. Techdirt has been an upstanding place for journalism forever, and getting censored this way is ridiculous.
- ziftface 3y ago
  You won't regret it, it's been a lot better than ddg for me.
- init2null 3y ago
  Kagi is an underrated superpower. It just works. I tried Google again recently, and I was astonished how mediocre the results were. If that's the gold standard for no-cost searching, the internet is in serious trouble.
  - alx__ 3y ago
    Yeah, Kagi returns better results for me. Is free, but totally worth the one coffee ($5) per month.
    - intrasight 3y ago
      Doesn't Kagi just use search APIs to get it results? Why would they be better if they don't build their own index?
      - alx__ 3y ago
        Where have you heard this? They build their own index: https://help.kagi.com/kagi/why-kagi/kagi-vs-competition.html https://help.kagi.com/kagi/why-kagi/kagi-vs-competition.html
        J_Shelby_J 3y ago
        They build their own index but don’t let you access it with keyword based queries. Instead the do ‘magic’ and shape the responses from multiple data sources. I’d love it if instead of ‘magic’ we just had a search engine that let you be the magician with a better query language and ux filters.
        VHRanger 3y ago
pdimitar 3y ago
Well, the last few months DDG's search quality started dropping hard for me, some queries that I clearly remember having good results no longer have them today. It seems there's stuff going on behind the scenes. DDG got taken over by some vested interest, perhaps. Or Bing doing stuff and DDG never branching out of it and just blindly getting results from it. Either way, it's starting to rival Google in uselessness and I'm likely to stop using it fairly soon. Kagi seems to work pretty well.
- yegg 3y ago
  Any particular examples you can remember that I can look into personally (and bring back to the team to look at as well)?
  - pdimitar 3y ago
    I apologize for not keeping them. Sorry. :( The thing with search engines is: you try one, scan results for 30 seconds, don't find a useful result, curse under your breath and hastily try another search engine. It's a normal programmer flow, which sadly almost completely eliminates the possibility to provide actionable feedback to the search engine's maintainers. I gather you're from the DDG team? If so, I'd say that the writing is on the wall that Bing is no longer a good backend for DDG. You guys should start branching out because from where I'm standing, many programmers are 10-15 annoyances away from switching. Again, my apologies for not providing examples.
    - Baeocystin 3y ago
      The person you're replying to is the CEO and founder of DDG. Just FYI.
      - pdimitar 3y ago
        Good to know, thanks. Had no clue.
  - worik 3y ago
aimor 3y ago
It would be neat to see a black list of all the things Bing and Google block from search results. I don't know how to get such a list other than brute force trial and error.
- registeredcorn 3y ago
  Better still would be a black list search engine to show only results that were banned by such services.
  - kccqzy 3y ago
    Then I guarantee your search engine will mostly contain spam, low-effort reposts, content farms, egregiously bad SEOs, and other useless stuff. There's a reason why mainstream search engines like DuckDuckGo perform this kind of blacklisting. Because 99.99% of the time this blacklisting only renders invisible what is worthless anyways. But that 0.01% of the time when it incorrectly blacklists a site HN is outraged.
    - registeredcorn 3y ago
      >Then I guarantee your search engine will mostly contain spam, low-effort reposts, content farms, egregiously bad SEOs, and other useless stuff. I can't help but think I've encountered this type of content before...but where. I also wouldn't exactly call this kind of response "outrage" but something close to befuddled amusement. "Whoops! Looks like we accidentally blocked those sites that criticize us! Yikes." If in err, then what an error! If by intent, then who would believe otherwise? I suppose some cranky, overly emotional weirdos might get in a tizzy, but this is pretty bog standard stuff. It's just fun to gawk at the sweaty fat man on the tricycle trying to backpedal across the motorway.
kermire 3y ago
Overall search quality has been declining on all search engines. Maybe there's too much spam. Saw an entertaining video about it yesterday that echoes how I feel when I google stuff: https://www.youtube.com/watch?v=jrFv1O4dbqY https://www.youtube.com/watch?v=jrFv1O4dbqY. It's so hard to find content written by humans these days. Seems like only the top sites are being indexed.
- CamperBob2 3y ago
  Google's attempt to introduce zero-click results has been nothing short of catastrophic. The sheer volume of bullshit they are spreading rivals anything that ChatGPT could ever hope to generate. A couple of weeks ago, I was debating with someone about what "LMR" stood for in the context of cable specifications, such as LMR-240, LMR-400 and so on. I thought it meant "Land Mobile Radio" while the other person disagreed that it stood for anything. A Google search on LMR coax cable acronym returned a helpful info blurb stating that LMR stood for "Last Minute Resistance" as a means of fending off sexual assault. Needless to say there was no way to tell exactly what site Google had copied that definition from, and no useful way to provide feedback to them. Sometimes there's a "Feedback" link, this time there wasn't. Sometimes the feedback link is present but only offers the option of reporting illegal activity. That option wasn't present either. For whatever reason, Google clearly does not give a flying fuck at a rolling donut about search quality anymore. With the right leadership, Bing could own that entire line of business, in a manner reminiscent of IE's original dominance over Netscape. I'm not holding my breath, but at this point I'm cheering for anyone who can offer Google some competition.
  - jes 3y ago
    I asked ChatGPT 4.0 what LMR meant as applied to coaxial cable. It gave me an excellent response, including that LMR is a trademark of Times Microwave Systems. I rarely use search engines anymore. I'll bet the same is true for many people.
    - lambic 3y ago
      I tend to ask ChatGPT first these days, but then follow up with a DDG search because I don't fully trust ChatGPT.
thewataccount 3y ago
I feel like this makes it clear the level of reliance DuckDuckGo has on Bing. I know it's been a bit ambigious the full extent to which they do, and they claim to have their own indexer, but there's no way this is a coincidence. Surely if you have your own indexer you would have seen techdirt before? At this moment "techdirt" only returns the wikipedia article, twitter, then mostly unrelated mentions of it. Surely DDG would have seen at least their homepage before? > Most of our search result pages feature one or more Instant Answers. To deliver Instant Answers on specific topics, DuckDuckGo leverages many sources, including specialized sources like Sportradar and crowd-sourced sites like Wikipedia. We also maintain our own crawler (DuckDuckBot) and many indexes to support our results. Of course, we have more traditional links and images in our search results too, which we largely source from Bing. Our focus is synthesizing all these sources to create a superior search experience. https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources...
- Brian_K_White 3y ago
  A dinky little but long standing staple of it's topic bitchin100.com had the same problem for some years but I guess the community was small enough and traditional enough that not enough people were using anything but google to notice except me. They eventually got it fixed but only after a few other people finally noticed and the owner contacted someone and then waited a week or so. But it was like that for years. You search a perfectly good search term that should pull up one of the articles on that site first, and all you got are all kinds of other 2nd and 3rd hand references like email archive posts and articles on other sites, but scroll down as many pages as you want and the actual site would never come up on ddg. That's when I got interested in Kagi.
- Retric 3y ago
  DDG’s business model is largely about privacy and UI rather than attempting to be objectively better than other search engines. Search is not quite 1:1 with Bing, but frankly I hate Bing’s UI not its results.
  - pb7 3y ago
WallyFunk 3y ago
Kagi[0] FTW [0] https://kagi.com/ https://kagi.com/
- deleted 3y ago
  [deleted]
- ehPReth 3y ago
  Is Kagi "streets ahead" of Google and the like? I find myself getting more and more frustrated at Google lately...
yegg 3y ago
(DuckDuckGo CEO/Founder) Just seeing this and we're looking into this now. This is not intentional. Update: Still investigating, but have made some progress -- Determined that on desktop there was a link to Techdirt up continuously via our About module (when you search for "Techdirt"). And now the traditional web link is back up as well (for desktop and mobile): https://duckduckgo.com/?q=techdirt&ia=web https://duckduckgo.com/?q=techdirt&ia=web.
- WheatMillington 3y ago
  I thought you barely relied on Bing anymore. At least, that's the claim you've repeatedly made here. This indicates that DDG is little more than a Bing mirror.
  - kats 3y ago
    [flagged]
  - yegg 3y ago
    I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807) but in short it isn't a binary thing and I think the fallacy people keep making is a modern search engine != web index. Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important. In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.
    - Waterluvian 3y ago
beej71 3y ago
Some of my stuff was in a Bing black hole for a while, and I did everything I could fix it, following their guidelines, contacting tech support, etc. It was good content, too--educational, no spam, SEO, ads, or tracking. Eventually I just gave up. Fast forward a year or so, and the search results spontaneously started working. So it's a black box, and you kind of just hope for the best.
tredre3 3y ago
That can't be right, DDG's CEO has assured me that DDG isn't just a dumb proxy to Bing. DDG uses several sources as well as its own index so it cannot suffer from what is being claimed [1][2][3]. 1. https://news.ycombinator.com/item?id=32360874 https://news.ycombinator.com/item?id=32360874 2. https://news.ycombinator.com/item?id=36149682 https://news.ycombinator.com/item?id=36149682 3. https://news.ycombinator.com/item?id=31492631 https://news.ycombinator.com/item?id=31492631
- WheatMillington 3y ago
  And yet here we are hmmmm
- yegg 3y ago
  None of that is contradictory. To paraphrase: - We have on the order of a million lines of search code at this point and have a lot of talented people working them. That code does a myriad of things across many indexes. - As an example, mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. We don't get any local search module content from Bing. - Similarly, on desktop, knowledge graph / Wikipedia-type answers come up the most and we don't get any of that module content from Bing either. - Bing is our largest source of traditional web links, which have become less and less relevant/engaged with over time as more and more modules are in search results and put on top of traditional links (and people interact with things on top of the search results page about two orders of magnitude vs. things on the bottom). - When Bing has dropped things out of the traditional web index, we have put them back, and we've been working with them so this happens less and less. In fact, there hasn't been hardly any reports of this in the past month or so, which is why I've asked for other examples in the comments.
  - giantrobot 3y ago
    As a long time DDG user I think it's cool you're personally engaged here. The disappointing part is it seems like TechDirt has had to resort to customer-service-via-social (Reddit/HN/Twitter) which sucks. Once this particular issue is sorted out I'd would also be cool to see some sort of post-mortem report. Seeing why this sort of thing happens and a standard process for fixing it would be beneficial to all involved I think. Complex systems are complex and shit happens. But if this happened to a site far smaller than TechDirt (they're not even that big AFAIK) I don't think there would be any avenue to cure the situation for them. /2¢
deleted 3y ago
[deleted]
xnx 3y ago
It's ridiculous to talk about DDG being separate from Google. In the same way Brave is Chrome with some minor doodads and settings, DDG is minor reskin of Bing.
- danudey 3y ago
  This is an inaccurate statement. See here for some specifics: https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807
vancan1ty 3y ago
I find that brave search is actually pretty good these days. Has plenty of results from techdirt at least... [1] https://search.brave.com/search?q=site%3Atechdirt.com&source=web https://search.brave.com/search?q=site%3Atechdirt.com&source...
riffic 3y ago
masnick must have pissed someone off recently lol
sproketboy 3y ago
[dead]
ty_2k 3y ago
It would be really nice if DDG had their own webmaster console (like Google or Bing) to help understand why pages might be indexed or dropped. I love DDG as my primary search engine, but it's a black box from the SEO side of the desk, unless I'm missing something obvious.
- FabHK 3y ago
  > [DDG is] a black box from the SEO side of the desk That’s a feature, not a bug, from my point of view.
- ipaddr 3y ago
  It doesn't index you, bing does.
photonerd 3y ago
Comments summary: lots of people who don’t understand how large search indexes work freaking out or pontificating about their pet favorite alternative. Occasionally both. Meanwhile the DDG founder calmly stating it’s being looked into, is definitely a bug, and explains that this should not generally happen… and being ignored.
- deleted 3y ago
  [deleted]
- bufferoverflow 3y ago
  Then how do you explain what we see? Tech Dirt disappears from both Bing and DDG indexes at the same time. Google, meanwhile, has 136 thousand pages indexed on techdirt.com domain
  - Choco31415 3y ago
    Like photonerd mentioned, yegg (The Founder of DDG) is answering that and similar questions in this thread: https://news.ycombinator.com/item?id=36898661 https://news.ycombinator.com/item?id=36898661
    - bufferoverflow 3y ago
      I don't see any answers on why tens of thousands of techdirt.com pages aren't in the DDG index. It's not some random unknown blog with zero traffic.
      - photonerd 3y ago
        Check his other comments in the replies. https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807 Meanwhile, thanks for illustrating my comment extraordinarily effectively, I guess?
        ipaddr 3y ago
NayamAmarshe 3y ago
Brave Search is doing pretty well. https://search.brave.com/search?q=techdirt https://search.brave.com/search?q=techdirt
- slig 3y ago
  Brave Search feels a lot like Google Search from back in the day.
lapcat 3y ago
This keeps happening. Just a few of many examples: https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-and-bing/ https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-a... https://www.jessesquires.com/blog/2022/03/25/my-website-disappeared-from-bing-and-duckduckgo/ https://www.jessesquires.com/blog/2022/03/25/my-website-disa... https://lapcatsoftware.com/articles/bing.html https://lapcatsoftware.com/articles/bing.html
KennyBlanken 3y ago
I find it simply impossible to believe that Masnick is so incompetent at running a website after so long, that he's finding out from random friends that his site has been de-listed from Bing instead of him or his staff having their search console lighting up like a dashboard and pinging them with emails. I get emails from both bing and google if there are site indexing issues, and while they're not always super clear about why, the major stuff is pretty easy to fix. I find it even harder to believe that when he confirmed the problem, he then didn't do anything about it for months. This feels incredibly performative. Hey Mike: instead of bing chat, try https://www.bing.com/webmaster/tools https://www.bing.com/webmaster/tools
butz 3y ago
And that's why we need a diverse ecosystem of search engines. As indexing whole internet is pretty much impossible, maybe several startups could take a slice of it? Some overlap would be useful too.
MrDresden 3y ago
This is unrelated to the issue at hand, but in case someone from Techdirt comes here then please, allow zoom on your site. There is no defensible reason to disable it on a content heavy site like yours.
flyinghamster 3y ago
It's looking like 2023 is the year that search dies. Between the removal of exclusion operators (hey, we saw you were adding -pinterest to your searches, but we want to make sure you get your pinterest!) and de-indexing of sites, it's looking like it's time for the search wheel to start making its third spin. Guess I need to start playing more with things like Algolia or Kagi.
- bufferoverflow 3y ago
  Unfortunately, creating a search engine competitive with Google or even Bing is insanely expensive. Not just from the server cost perspective, but also from marketing it and convincing people to switch.
jdjdjdhhd 3y ago
DDG is useless, unless you really like Bing and care a bit about privacy... I'd rather use Brave for search and Yandex for image search.
thriftwy 3y ago
Just checked - Yandex has recent pages from techdirt.com in search results.
mutant_glofish 3y ago
DuckDuckGo has a history of inheriting censorship from Bing [1] and also doing it on its own. [2] [1]: https://reclaimthenet.org/microsofts-bing-censors-tank-man https://reclaimthenet.org/microsofts-bing-censors-tank-man [2]: https://www.washingtonexaminer.com/news/duckduckgo-slammed-for-downrating-russian-search-results https://www.washingtonexaminer.com/news/duckduckgo-slammed-f...
oidar 3y ago
techdirt is working fine on Kagi. Ever since its launch, I've been predominantly using Kagi, barely resorting to any other search engine. I estimate that only about 5% of my searches involve Google, and that's mainly when I need to utilize Scholar or Books. Kagi is definately worth the $10 fee. First, it allows me to block websites that I find irrelevant or annoying in my areas of interest, such as Pinterest or Quora. It also promises an environment free of advertisements and tracking. I particularly appreciate the excellent customer support, where actual human beings respond to your emails, not automated responses. Additionally, it offers 'custom lenses', which are essentially search templates. For instance, using this feature I can refine my search to target only educational sites, filter for PDFs, limit the time to the "last 48 hours", and search for specific subjects, like machine learning, along with my query. My only gripe with Kagi is its stringent limit on searches, set at around 10,000 per month. I've bumped against this ceiling a few times. Despite this limitation, the wealth of features and the quality of the search results make Kagi a worthwhile investment for me.
phreack 3y ago
Title says [fixed] but that's only for DDG - which despite all the flaming it's getting, somehow did fix it while Bing has not (will it?). My unfounded theory is that Techdirt is effectively banned on some country and that leaked to the global Bing index. https://www.bing.com/search?q=site%3Atechdirt.com https://www.bing.com/search?q=site%3Atechdirt.com
- sct202 3y ago
  It doesn't look fixed to me. Only the homepage shows up on DDG from searching site:techdirt.com Google has 100k+ results for the same prompt.
  - AshamedCaptain 3y ago
    Most glaring example is that looking for this headline, as of right now, does not return any Techdirt link, but does return HN and reddit links. https://duckduckgo.com/?q=%22Techdirt+Has+Been+Deleted+From+Bing%22&ia=web https://duckduckgo.com/?q=%22Techdirt+Has+Been+Deleted+From+...
davb 3y ago
Whatever the complaints about DDG, I still commend yegg for engaging here. I can’t imagine the head of Google Search or Bing doing the same.
philipov 3y ago
When I do the search on bing, it doesn't just give no results, it also says "Some results have been removed" with a link to their content policy page.
maxlin 3y ago
After DDG's earlier announcements delivering the image that DDG then severed their uneasy dependency from Bing, this makes them look really bad. Retorting with mumbo jumbo about secondary stuff like flights, business listings etc that are just related to website retention rather than their core structure as an apparent search engine, just makes it worse. I still believe DDG has a place, in being a more privacy-focused aggregator to Bing and a few other sources, but this did shatter the previous image of DDG having a large degree of autonomy for me.

220 points by lehi 3y ago

HWR_14 3y ago

I've noticed DDG dropping a lot of sites over the past month or so.

yegg 3y ago
Do you know what they are? Happy to investigate along with TechDirt and that might help.
- HWR_14 3y ago
  I don't recall at the moment. If I remember I'll let you know.
  - mdaniel 3y ago
    a more sustainable approach would be to use the "Share Feedback" in the (regrettably positioned) lower right corner of the search results page. The story I've heard is that actual humans look at that feedback
    - yegg 3y ago
      Yes, that's right -- we do!

VHRanger 3y ago

As a long time DDG user, I was looking for a reason to try Kagi out, and this nonsense is finally pushing me over the fence. Techdirt has been an upstanding place for journalism forever, and getting censored this way is ridiculous.

ziftface 3y ago
You won't regret it, it's been a lot better than ddg for me.
init2null 3y ago
Kagi is an underrated superpower. It just works. I tried Google again recently, and I was astonished how mediocre the results were. If that's the gold standard for no-cost searching, the internet is in serious trouble.
- alx__ 3y ago
  Yeah, Kagi returns better results for me. Is free, but totally worth the one coffee ($5) per month.
  - intrasight 3y ago
    Doesn't Kagi just use search APIs to get it results? Why would they be better if they don't build their own index?
    - alx__ 3y ago
      Where have you heard this? They build their own index: https://help.kagi.com/kagi/why-kagi/kagi-vs-competition.html https://help.kagi.com/kagi/why-kagi/kagi-vs-competition.html
      - J_Shelby_J 3y ago
        They build their own index but don’t let you access it with keyword based queries. Instead the do ‘magic’ and shape the responses from multiple data sources. I’d love it if instead of ‘magic’ we just had a search engine that let you be the magician with a better query language and ux filters.
        VHRanger 3y ago

pdimitar 3y ago

Well, the last few months DDG's search quality started dropping hard for me, some queries that I clearly remember having good results no longer have them today. It seems there's stuff going on behind the scenes. DDG got taken over by some vested interest, perhaps. Or Bing doing stuff and DDG never branching out of it and just blindly getting results from it. Either way, it's starting to rival Google in uselessness and I'm likely to stop using it fairly soon. Kagi seems to work pretty well.

yegg 3y ago
Any particular examples you can remember that I can look into personally (and bring back to the team to look at as well)?
- pdimitar 3y ago
  I apologize for not keeping them. Sorry. :( The thing with search engines is: you try one, scan results for 30 seconds, don't find a useful result, curse under your breath and hastily try another search engine. It's a normal programmer flow, which sadly almost completely eliminates the possibility to provide actionable feedback to the search engine's maintainers. I gather you're from the DDG team? If so, I'd say that the writing is on the wall that Bing is no longer a good backend for DDG. You guys should start branching out because from where I'm standing, many programmers are 10-15 annoyances away from switching. Again, my apologies for not providing examples.
  - Baeocystin 3y ago
    The person you're replying to is the CEO and founder of DDG. Just FYI.
    - pdimitar 3y ago
      Good to know, thanks. Had no clue.
- worik 3y ago

aimor 3y ago

It would be neat to see a black list of all the things Bing and Google block from search results. I don't know how to get such a list other than brute force trial and error.

registeredcorn 3y ago
Better still would be a black list search engine to show only results that were banned by such services.
- kccqzy 3y ago
  Then I guarantee your search engine will mostly contain spam, low-effort reposts, content farms, egregiously bad SEOs, and other useless stuff. There's a reason why mainstream search engines like DuckDuckGo perform this kind of blacklisting. Because 99.99% of the time this blacklisting only renders invisible what is worthless anyways. But that 0.01% of the time when it incorrectly blacklists a site HN is outraged.
  - registeredcorn 3y ago
    >Then I guarantee your search engine will mostly contain spam, low-effort reposts, content farms, egregiously bad SEOs, and other useless stuff. I can't help but think I've encountered this type of content before...but where. I also wouldn't exactly call this kind of response "outrage" but something close to befuddled amusement. "Whoops! Looks like we accidentally blocked those sites that criticize us! Yikes." If in err, then what an error! If by intent, then who would believe otherwise? I suppose some cranky, overly emotional weirdos might get in a tizzy, but this is pretty bog standard stuff. It's just fun to gawk at the sweaty fat man on the tricycle trying to backpedal across the motorway.

kermire 3y ago

Overall search quality has been declining on all search engines. Maybe there's too much spam. Saw an entertaining video about it yesterday that echoes how I feel when I google stuff: https://www.youtube.com/watch?v=jrFv1O4dbqY https://www.youtube.com/watch?v=jrFv1O4dbqY. It's so hard to find content written by humans these days. Seems like only the top sites are being indexed.

CamperBob2 3y ago
Google's attempt to introduce zero-click results has been nothing short of catastrophic. The sheer volume of bullshit they are spreading rivals anything that ChatGPT could ever hope to generate. A couple of weeks ago, I was debating with someone about what "LMR" stood for in the context of cable specifications, such as LMR-240, LMR-400 and so on. I thought it meant "Land Mobile Radio" while the other person disagreed that it stood for anything. A Google search on LMR coax cable acronym returned a helpful info blurb stating that LMR stood for "Last Minute Resistance" as a means of fending off sexual assault. Needless to say there was no way to tell exactly what site Google had copied that definition from, and no useful way to provide feedback to them. Sometimes there's a "Feedback" link, this time there wasn't. Sometimes the feedback link is present but only offers the option of reporting illegal activity. That option wasn't present either. For whatever reason, Google clearly does not give a flying fuck at a rolling donut about search quality anymore. With the right leadership, Bing could own that entire line of business, in a manner reminiscent of IE's original dominance over Netscape. I'm not holding my breath, but at this point I'm cheering for anyone who can offer Google some competition.
- jes 3y ago
  I asked ChatGPT 4.0 what LMR meant as applied to coaxial cable. It gave me an excellent response, including that LMR is a trademark of Times Microwave Systems. I rarely use search engines anymore. I'll bet the same is true for many people.
  - lambic 3y ago
    I tend to ask ChatGPT first these days, but then follow up with a DDG search because I don't fully trust ChatGPT.

thewataccount 3y ago

I feel like this makes it clear the level of reliance DuckDuckGo has on Bing. I know it's been a bit ambigious the full extent to which they do, and they claim to have their own indexer, but there's no way this is a coincidence. Surely if you have your own indexer you would have seen techdirt before? At this moment "techdirt" only returns the wikipedia article, twitter, then mostly unrelated mentions of it. Surely DDG would have seen at least their homepage before? > Most of our search result pages feature one or more Instant Answers. To deliver Instant Answers on specific topics, DuckDuckGo leverages many sources, including specialized sources like Sportradar and crowd-sourced sites like Wikipedia. We also maintain our own crawler (DuckDuckBot) and many indexes to support our results. Of course, we have more traditional links and images in our search results too, which we largely source from Bing. Our focus is synthesizing all these sources to create a superior search experience. https://duckduckgo.com/duckduckgo-help-pages/results/sources/ https://duckduckgo.com/duckduckgo-help-pages/results/sources...

Brian_K_White 3y ago
A dinky little but long standing staple of it's topic bitchin100.com had the same problem for some years but I guess the community was small enough and traditional enough that not enough people were using anything but google to notice except me. They eventually got it fixed but only after a few other people finally noticed and the owner contacted someone and then waited a week or so. But it was like that for years. You search a perfectly good search term that should pull up one of the articles on that site first, and all you got are all kinds of other 2nd and 3rd hand references like email archive posts and articles on other sites, but scroll down as many pages as you want and the actual site would never come up on ddg. That's when I got interested in Kagi.
Retric 3y ago
DDG’s business model is largely about privacy and UI rather than attempting to be objectively better than other search engines. Search is not quite 1:1 with Bing, but frankly I hate Bing’s UI not its results.
- pb7 3y ago

WallyFunk 3y ago

Kagi[0] FTW [0] https://kagi.com/ https://kagi.com/

deleted 3y ago
[deleted]
ehPReth 3y ago
Is Kagi "streets ahead" of Google and the like? I find myself getting more and more frustrated at Google lately...

yegg 3y ago

(DuckDuckGo CEO/Founder) Just seeing this and we're looking into this now. This is not intentional. Update: Still investigating, but have made some progress -- Determined that on desktop there was a link to Techdirt up continuously via our About module (when you search for "Techdirt"). And now the traditional web link is back up as well (for desktop and mobile): https://duckduckgo.com/?q=techdirt&ia=web https://duckduckgo.com/?q=techdirt&ia=web.

WheatMillington 3y ago
I thought you barely relied on Bing anymore. At least, that's the claim you've repeatedly made here. This indicates that DDG is little more than a Bing mirror.
- kats 3y ago
  [flagged]
- yegg 3y ago
  I don't believe I have ever made that claim and it isn't true either that we are "little more than a Bing mirror". See other comment on this thread (https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807) but in short it isn't a binary thing and I think the fallacy people keep making is a modern search engine != web index. Over the past fifteen years, search has become "universal" with dozens of indexes and modules throughout the page. Also it is important to understand people click/engage with things on the page roughly half as much each position down, so by the time you get to the bottom it is like 100 times less than the top. This means that things on top, increasingly non-web links, have become more and more important. In this context, as mentioned in the other comment, the largest modules on mobile and desktop we power ourselves, that is, local and knowledge graph. AI will be the same. We do use Bing as the primary source for traditional web links, but not for all and even when we do it sometimes looks different in various ways. In something like this, we can re-insert this link if that is what is needed.
  - Waterluvian 3y ago

beej71 3y ago

Some of my stuff was in a Bing black hole for a while, and I did everything I could fix it, following their guidelines, contacting tech support, etc. It was good content, too--educational, no spam, SEO, ads, or tracking. Eventually I just gave up. Fast forward a year or so, and the search results spontaneously started working. So it's a black box, and you kind of just hope for the best.

tredre3 3y ago

That can't be right, DDG's CEO has assured me that DDG isn't just a dumb proxy to Bing. DDG uses several sources as well as its own index so it cannot suffer from what is being claimed [1][2][3]. 1. https://news.ycombinator.com/item?id=32360874 https://news.ycombinator.com/item?id=32360874 2. https://news.ycombinator.com/item?id=36149682 https://news.ycombinator.com/item?id=36149682 3. https://news.ycombinator.com/item?id=31492631 https://news.ycombinator.com/item?id=31492631

WheatMillington 3y ago
And yet here we are hmmmm
yegg 3y ago
None of that is contradictory. To paraphrase: - We have on the order of a million lines of search code at this point and have a lot of talented people working them. That code does a myriad of things across many indexes. - As an example, mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. We don't get any local search module content from Bing. - Similarly, on desktop, knowledge graph / Wikipedia-type answers come up the most and we don't get any of that module content from Bing either. - Bing is our largest source of traditional web links, which have become less and less relevant/engaged with over time as more and more modules are in search results and put on top of traditional links (and people interact with things on top of the search results page about two orders of magnitude vs. things on the bottom). - When Bing has dropped things out of the traditional web index, we have put them back, and we've been working with them so this happens less and less. In fact, there hasn't been hardly any reports of this in the past month or so, which is why I've asked for other examples in the comments.
- giantrobot 3y ago
  As a long time DDG user I think it's cool you're personally engaged here. The disappointing part is it seems like TechDirt has had to resort to customer-service-via-social (Reddit/HN/Twitter) which sucks. Once this particular issue is sorted out I'd would also be cool to see some sort of post-mortem report. Seeing why this sort of thing happens and a standard process for fixing it would be beneficial to all involved I think. Complex systems are complex and shit happens. But if this happened to a site far smaller than TechDirt (they're not even that big AFAIK) I don't think there would be any avenue to cure the situation for them. /2¢

deleted 3y ago

[deleted]

xnx 3y ago

It's ridiculous to talk about DDG being separate from Google. In the same way Brave is Chrome with some minor doodads and settings, DDG is minor reskin of Bing.

danudey 3y ago
This is an inaccurate statement. See here for some specifics: https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807

vancan1ty 3y ago

I find that brave search is actually pretty good these days. Has plenty of results from techdirt at least... [1] https://search.brave.com/search?q=site%3Atechdirt.com&source=web https://search.brave.com/search?q=site%3Atechdirt.com&source...

riffic 3y ago

masnick must have pissed someone off recently lol

sproketboy 3y ago

[dead]

ty_2k 3y ago

It would be really nice if DDG had their own webmaster console (like Google or Bing) to help understand why pages might be indexed or dropped. I love DDG as my primary search engine, but it's a black box from the SEO side of the desk, unless I'm missing something obvious.

FabHK 3y ago
> [DDG is] a black box from the SEO side of the desk That’s a feature, not a bug, from my point of view.
ipaddr 3y ago
It doesn't index you, bing does.

photonerd 3y ago

Comments summary: lots of people who don’t understand how large search indexes work freaking out or pontificating about their pet favorite alternative. Occasionally both. Meanwhile the DDG founder calmly stating it’s being looked into, is definitely a bug, and explains that this should not generally happen… and being ignored.

deleted 3y ago
[deleted]
bufferoverflow 3y ago
Then how do you explain what we see? Tech Dirt disappears from both Bing and DDG indexes at the same time. Google, meanwhile, has 136 thousand pages indexed on techdirt.com domain
- Choco31415 3y ago
  Like photonerd mentioned, yegg (The Founder of DDG) is answering that and similar questions in this thread: https://news.ycombinator.com/item?id=36898661 https://news.ycombinator.com/item?id=36898661
  - bufferoverflow 3y ago
    I don't see any answers on why tens of thousands of techdirt.com pages aren't in the DDG index. It's not some random unknown blog with zero traffic.
    - photonerd 3y ago
      Check his other comments in the replies. https://news.ycombinator.com/item?id=36898807 https://news.ycombinator.com/item?id=36898807 Meanwhile, thanks for illustrating my comment extraordinarily effectively, I guess?
      - ipaddr 3y ago

NayamAmarshe 3y ago

Brave Search is doing pretty well. https://search.brave.com/search?q=techdirt https://search.brave.com/search?q=techdirt

slig 3y ago
Brave Search feels a lot like Google Search from back in the day.

lapcat 3y ago

This keeps happening. Just a few of many examples: https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-and-bing/ https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-a... https://www.jessesquires.com/blog/2022/03/25/my-website-disappeared-from-bing-and-duckduckgo/ https://www.jessesquires.com/blog/2022/03/25/my-website-disa... https://lapcatsoftware.com/articles/bing.html https://lapcatsoftware.com/articles/bing.html

KennyBlanken 3y ago

I find it simply impossible to believe that Masnick is so incompetent at running a website after so long, that he's finding out from random friends that his site has been de-listed from Bing instead of him or his staff having their search console lighting up like a dashboard and pinging them with emails. I get emails from both bing and google if there are site indexing issues, and while they're not always super clear about why, the major stuff is pretty easy to fix. I find it even harder to believe that when he confirmed the problem, he then didn't do anything about it for months. This feels incredibly performative. Hey Mike: instead of bing chat, try https://www.bing.com/webmaster/tools https://www.bing.com/webmaster/tools

butz 3y ago

And that's why we need a diverse ecosystem of search engines. As indexing whole internet is pretty much impossible, maybe several startups could take a slice of it? Some overlap would be useful too.

MrDresden 3y ago

This is unrelated to the issue at hand, but in case someone from Techdirt comes here then please, allow zoom on your site. There is no defensible reason to disable it on a content heavy site like yours.

flyinghamster 3y ago

It's looking like 2023 is the year that search dies. Between the removal of exclusion operators (hey, we saw you were adding -pinterest to your searches, but we want to make sure you get your pinterest!) and de-indexing of sites, it's looking like it's time for the search wheel to start making its third spin. Guess I need to start playing more with things like Algolia or Kagi.

bufferoverflow 3y ago
Unfortunately, creating a search engine competitive with Google or even Bing is insanely expensive. Not just from the server cost perspective, but also from marketing it and convincing people to switch.

jdjdjdhhd 3y ago

DDG is useless, unless you really like Bing and care a bit about privacy... I'd rather use Brave for search and Yandex for image search.

thriftwy 3y ago

Just checked - Yandex has recent pages from techdirt.com in search results.

mutant_glofish 3y ago

DuckDuckGo has a history of inheriting censorship from Bing [1] and also doing it on its own. [2] [1]: https://reclaimthenet.org/microsofts-bing-censors-tank-man https://reclaimthenet.org/microsofts-bing-censors-tank-man [2]: https://www.washingtonexaminer.com/news/duckduckgo-slammed-for-downrating-russian-search-results https://www.washingtonexaminer.com/news/duckduckgo-slammed-f...

oidar 3y ago

techdirt is working fine on Kagi. Ever since its launch, I've been predominantly using Kagi, barely resorting to any other search engine. I estimate that only about 5% of my searches involve Google, and that's mainly when I need to utilize Scholar or Books. Kagi is definately worth the $10 fee. First, it allows me to block websites that I find irrelevant or annoying in my areas of interest, such as Pinterest or Quora. It also promises an environment free of advertisements and tracking. I particularly appreciate the excellent customer support, where actual human beings respond to your emails, not automated responses. Additionally, it offers 'custom lenses', which are essentially search templates. For instance, using this feature I can refine my search to target only educational sites, filter for PDFs, limit the time to the "last 48 hours", and search for specific subjects, like machine learning, along with my query. My only gripe with Kagi is its stringent limit on searches, set at around 10,000 per month. I've bumped against this ceiling a few times. Despite this limitation, the wealth of features and the quality of the search results make Kagi a worthwhile investment for me.

phreack 3y ago

Title says [fixed] but that's only for DDG - which despite all the flaming it's getting, somehow did fix it while Bing has not (will it?). My unfounded theory is that Techdirt is effectively banned on some country and that leaked to the global Bing index. https://www.bing.com/search?q=site%3Atechdirt.com https://www.bing.com/search?q=site%3Atechdirt.com

sct202 3y ago
It doesn't look fixed to me. Only the homepage shows up on DDG from searching site:techdirt.com Google has 100k+ results for the same prompt.
- AshamedCaptain 3y ago
  Most glaring example is that looking for this headline, as of right now, does not return any Techdirt link, but does return HN and reddit links. https://duckduckgo.com/?q=%22Techdirt+Has+Been+Deleted+From+Bing%22&ia=web https://duckduckgo.com/?q=%22Techdirt+Has+Been+Deleted+From+...

davb 3y ago

Whatever the complaints about DDG, I still commend yegg for engaging here. I can’t imagine the head of Google Search or Bing doing the same.

philipov 3y ago

When I do the search on bing, it doesn't just give no results, it also says "Some results have been removed" with a link to their content policy page.

maxlin 3y ago

After DDG's earlier announcements delivering the image that DDG then severed their uneasy dependency from Bing, this makes them look really bad. Retorting with mumbo jumbo about secondary stuff like flights, business listings etc that are just related to website retention rather than their core structure as an apparent search engine, just makes it worse. I still believe DDG has a place, in being a more privacy-focused aggregator to Bing and a few other sources, but this did shatter the previous image of DDG having a large degree of autonomy for me.