15 ms·
I'm shadow banned by DuckDuckGo and Bing
- fruit2020 4y agoTangential question to the site index. How does one get the TLD and ccTLD zone data, apparently it’s not that open. There are some ccTLDs which give this info, for example .ch.
- supermatt 4y agoThe Centralized Zone Data Service: https://czds.icann.org https://czds.icann.org
- deleted 4y ago[deleted]
- Jakob 4y agoI asked myself the same thing last week and published a how to, like the sibling post says, the czds is the way: https://www.jakobstoeck.de/2023/download-and-query-all-domains-of-the-world-wide-web/ https://www.jakobstoeck.de/2023/download-and-query-all-domai...
- henriquez 4y agoDid you perhaps abbreviate Microsoft as M$ (with a dollar sign)-? That really pisses them off.
- supermatt 4y agoThis is not a shadow ban. A shadow ban is when it appears to you that you are not banned, but from others perspective you are.
- tasuki 4y agoYes. Unfortunately people increasingly use "shadow ban" to just mean "ban", perhaps it sounds cool?
- cactusplant7374 4y agoElon wants to redefine the term as well but for the purpose of a coordinated witch hunt. People like using shadow ban because it sounds more malicious vs. content that is no longer actively promoted by a company. It's hard to claim you're a victim if the reason is you're just not that interesting or popular.
- extheat 4y agoWell if you are silently placed on a hide list (by manual human intervention without you being able to know without a third party), then by all means that is a shadow ban. Edit for clarity.
- causality0 4y ago"Shadow ban" would be a good term to use for that if it didn't already have a meaning that was different. But it does.
- extheat 4y agoHence why I said silently. If I’m posting something and it appears to me as usual on my feed but not others, by all means that’s a shadow ban.
- InspiredIdiot 4y agoNope. Still wrong. I understand there is some way it feels like that from your perspective but DDG and Bing don't own your feed. So they are hiding nothing from you. They are 100% ~up-front~ (edit) consistent about not choosing to show your site (ban it) and the fact that they don't control your feed doesn't make it any more accurate to apply the word "shadow" to their ban.
- InspiredIdiot 4y agoI have become what I hate most. Someone who didn't read the article. I've heard too many instances of the term being misapplied and jumped to a conclusion based on the discussion. If you use Bing webmaster tools (a logged-in account for the use of the domain owner/content creator) and you can see indications that Bing indexed the content and no indications that errors preventing it from being eligible to show then it is certainly at least closer to a shadow ban than I originally thought. Still, any reference to a "feed" entirely misses the point unless Bing is the one also serving that feed. I can't see any evidence that Bing displays a feed to the poster.
- rapnie 4y agoIs there also a term for being algorithmically suppressed on a social media platform, I wonder? I.e. a much more subtle, harder to dectect mechanism whereby the algorithms ensure you get some exposure, but never the same as other unsuppressed people would get based on similar activity. Or only exposure to a certain limited subset of the graph based on some metrics (e.g. just your 'friends' so no one points out you are effectively shadow banned).
- Marazan 4y agoYes, its called Algorithmic Supression. That doesn't sound as cool and victimey though.
- spoiler 4y agoHow about Algorithmic Oppression
- jokowueu 4y agoAlgoOped
- cactusplant7374 4y agoIf the algorithm is actually a true measure of engagement or usefulness of your site is it just plain 'ol search engine justice?
- rapnie 4y agoCould call it a RoboBlock in popular language, with reference to a roadblock. An obstruction to engage enforced by our machine overlords. Or better maybe, a RoboGag.
- colanderman 4y ago"Automatically downranked/downweighted/penalized" are terms I've heard.
- hutzlibu 4y agoI think unlike in the case here, (soft) shadow banning would be appropriate to describe it, even though not 100% technical correct.
- ffhhj 4y agoThe author is extending the concept to include an inadverted ban. Why would Bing warn him anyways, since there is no user account? Welcome to Cancelbannia.
- seanhunter 4y agoIndeed. Some people seem to be trying to make it apply any time the top result of an algorithmic ranking isn't what they think it should be (eg if they have been deboosted rather than shadow banned).
- Dylan16807 4y agoI think being banned from search could be part of a shadow ban, but when the entire service is search that's just a ban.
- mkl 4y agoIf it was a shadow ban he would see his site when he searched but we wouldn't. This is definitely not that.
- Dylan16807 4y agoAre you talking about normal sites? Not search engines? If so I don't really agree. I don't think you have to go through that much effort as part of implementing a shadow ban. If posts show up to the user in most places, then good enough it's a shadow ban. Only because this is a search engine do I say this solidly isn't a shadow ban.
- mkl 4y agoWell, I don't understand the distinction you're making, as search doesn't seem special here to me. The key feature that makes something a shadow ban is that everything looks normal to the shadow-banned person, but the rest of us don't see their stuff or see it in some restricted way. That seems like it would apply equally to search, forum comments, etc.
- deleted 4y ago[deleted]
- JadoJodo 4y agoI suspect OP used it in the way that “I believe all my posts are showing up in the places that they should, but unbeknownst to me, they are being suppressed.” In this instance, I could see a “search engine shadowban” being an appropriate moniker.
- InspiredIdiot 4y agoI think a good test of whether this application of the term makes any sense is: Could any search engine ban ever not be a shadow ban? We already have a term: ban. Let's just use that one and stop conflating things and being unnecessary imprecise and incendiary. It helps certain parties' (edit plural possessive) agenda but does not help us clearly communicate.
- Eleison23 4y ago[dead]
- travisgriggs 4y agoClearly we need a new term. I nominate Ghost Banned or Ghostdexed
- stephencanon 4y agoOutdexed, clearly.
- zxcvbn4038 4y agoCheck with Stanford first, “ghost” might be offensive to the living impaired. =P
- _8j50 4y agoThat is what is happening, you think you are not showing up because bad SEO or better results. You have to find out through experimentatiom that you are restricted. The moderator didn't let you know that they have taken punitive action against you.
- supermatt 4y agoThat would just be a ban. You dont need to be told you are banned.
- sergiomattei 4y agoPerhaps you haven’t been indexed yet?
- freitasm 4y agoThe blog seems to be going since at least 2009. Long enough to be indexed. Perhaps some server-side filtering going on, blocking the bingbot user-agent?
- deleted 4y ago[deleted]
- mananaysiempre 4y agoFor some reason, Beej’s Guide to C Programming is also banned from Bing (and consequently DDG) [1], with the standard robotic non-explanations given when the author asked, even though the rest of the site is not. [1] https://beej.us/guide/bgc/whynoddg.html https://beej.us/guide/bgc/whynoddg.html
- Pelam 4y agoI can find Beej.us with DDG, but not daverupert.com. Maybe Beej got the problem resolved somehow.
- mananaysiempre 4y agoHe says in the link it’s specifically the C guide, the rest of the website is fine. Though... yeah, DDG queries like “beej c guide strlen” give reasonable results for me, if with an unjustifiably high-ranked position for the mirror at http://docs.hfbk.net/beej.us http://docs.hfbk.net/beej.us. Bing ones only include the mirror and the other guides (and a Scribd-hosted PDF copy, of all things, as the first result below a huge navigation card referring to https://beej.us/guides https://beej.us/guides but without the C guide among the links).
- eps 4y agohttps://beej.us/guide/bgclr/ https://beej.us/guide/bgclr/ is the 2nd result for "beej's c", with first being site's homepage.
- beej71 4y agoAh, I was just wondering if splitting the book in two volumes (900 pages is more than Amazon can print!) would impact this. But it still doesn't index the first volume...? Thanks for the info.
- projektfu 4y agoIncidentally, what is the "legal status" of Scribd hosting a partial preview of works that they tell you are BY-NC (attribution/non-commercial use only) and telling you to become a member to be able to view the whole thing? Is that not a commercial use?
- deleted 4y ago[deleted]
- donatj 4y agoThis triggered me to DuckDuckGo my own site and immediately I notice the top result is someone rehosting my OSS on a page loaded with pages of crap SEO content. Scrolling further, I don’t seem to find my own site either… https://donatstudios.com https://donatstudios.com I’ve added my site into Bing webmaster tools, we’ll see if it helps I guess.
- HomeDeLaPot 4y agoWow. I think I've used your circle generator on that other site without realizing it. That sucks. I've bookmarked yours now!
- donatj 4y agoA bunch of sites popped up hosting it loaded with SEO garbage and ads. I’d licensed it MIT so while they certainly can, it sure doesn’t feel nice.
- tinus_hn 4y agoSame thing happened with StackExchange and Wikipedia, but they appear to have fixed it.
- esperent 4y agoI've writing and freely shared a lot of OSS code and I plan to continue doing that. All MIT licensed. But I keep thinking that this is a limitation of MIT. It's written with the purest of good intentions but without any way to prevent bad actors from exploiting that. I wonder if there's a better kind of license that provides a close level of freedom but would prevent some of the most obvious exploitations (e.g. packaging and selling the code, or reposting on a site with advertising)? There's copyleft licensing like LGPL but perhaps that goes too far in the other direction and besides, it seems to be very unpopular amongst web developers so I'm afraid if I release any LGPL code it won't get used much or attract contributors. Is there a happy medium between these?
- 4y ago
- mg 4y agoLooking around his page, I see some interesting aspects on his atom page: https://daverupert.com/atom.xml https://daverupert.com/atom.xml First, he sends it with a "content-type: application/xml" header. In contrast to most sites that send it with "content-type: application/atom+xml". Which seems to have the nice effect that it renders in Firefox instead of opening the usual "What should Firefox do with this file?" popup. Secondly, he provides this nice header text "Yahaha, you found me! This is my RSS feed.". It seems to be fetched via this part of the code: <?xml-stylesheet href="/pretty-feed-v3.xsl" type="text/xsl"?> Pretty nice. Are those best practices? Or will "content-type: application/xml" mess with users who have a native feed reader installed and expect the reader to kick in when they click on a feed url?
- rzzzt 4y agoXSLT is a template language where you match snippets of XML using XPath expressions and output... anything else, in this case HTML using templates that can make use of the attributes, inner content, etc. of the captured XML portion. "An elegant weapon for a more... civilized age."
- Semaphor 4y ago> "An elegant weapon for a more... civilized age." One of the ideas people had back then was having a product catalog in XML that your tool would give you, and that you could upload to your website and display as a nice website via XSLT. It was such a fascinating technology when I learned about it, but I’m not sure if it ever saw any serious use? Maybe in enterprise?
- jbotdev 4y agoIt’s still used by the SEC to render company filings on their EDGAR site. I can’t find the exact files/docs, but I’ve rendered filings like this using the XML and XSLT they provide: https://www.sec.gov/ix?doc=/Archives/edgar/data/320193/000032019322000108/aapl-20220924.htm https://www.sec.gov/ix?doc=/Archives/edgar/data/320193/00003...
- 4y ago
- pseudolus 4y agoYou've indicated that you've used Bing's tools to see if your website has been indexed but are silent as to whether you've actually manually submitted your site to be indexed by Bing using their url submission tool [0]. If you do submit the URL and then, after a decent interval, your site still doesn't show up then there might be something to your claim. [0] https://www.bing.com/webmasters/help/url-submission-62f2860b https://www.bing.com/webmasters/help/url-submission-62f2860b
- omgmajk 4y agoAt the bottom of the page there is a comment that says "Some results have been removed", but unlike Google you can't see them. Would be interesting to know if the domain is among the removed results or if the site has not just been indexed yet.
- sct202 4y agoYou can submit a ticket to bing via their webmaster tools website. I've done it before in the past and a real human did respond at the time. In my experience bing will straight deindex full websites for unknown reasons while Google will ranking penalize but leave you searchable if their algo feels you deserve it.
- linmob 4y agoDid you manage to get re-indexed? I did not, and after the first attempt, replies to my tickets became quite rare.
- marginalia_nu 4y agoI got you, fam. https://search.marginalia.nu/site/daverupert.com https://search.marginalia.nu/site/daverupert.com
- TomK32 4y agoThe W3 HTML Validator is complaining about a few things, not sure which one could make bing think the website is not worth indexing, but it's worth a shot fixing those issues. https://validator.w3.org/nu/?doc=https%3A%2F%2Fdaverupert.com%2F https://validator.w3.org/nu/?doc=https%3A%2F%2Fdaverupert.co...
- lapcat 4y agoSee "Bing and DuckDuckGo removed my business web site" https://lapcatsoftware.com/articles/bing.html https://lapcatsoftware.com/articles/bing.html and "My website disappeared from Bing and DuckDuckGo, Part 2" https://www.jessesquires.com/blog/2022/07/25/my-website-disappeared-from-bing-and-duckduckgo-part-2/ https://www.jessesquires.com/blog/2022/07/25/my-website-disa... [EDIT] I just published a new blog post "Bing and DuckDuckGo removed my business web site AGAIN" https://lapcatsoftware.com/articles/bing2.html https://lapcatsoftware.com/articles/bing2.html Sigh.
- teekert 4y agoM$ is such a black box. At some point getting on their spam list, for no reason and still I’m not 100% certain, is the reason I stopped self hosting email.
- reaperducer 4y agoBlame SpamCop. I recently discovered that one of my services is on stage MS email naughty list, and found out it's because MS uses SpamCop. I contacted SpamCop and it was very responsive. Unfortunately, the solution SpamCop suggested was to move the entire project to a different provider.
- Joel_Mckay 4y agoNegative SEO scammers use intentional search-policy violations to push down the rank of perceived competitors for a few weeks. While it is more likely the poster will get a few people to check on the situation and naively drive up page rank... a personal site is just a rounding error for traffic in a long-tail distribution known as the modern web. Most search engines will correlate user-side telemetry traffic against crawler and web stats. i.e. if the bots tend to prefer your site for abnormal reasons, the ranking algorithm may blacklist a signature, domain, and IP sets for several weeks as punishment. Note too, it is still common for a human employee to manually check a suddenly popular site that pops up out of obscurity. i.e. this catches the more sophisticated cheats, and may have legal repercussions in severe cases. In summary, if you mess with modern search engines, than expect the ban hammer to fall eventually. ;)
- Anon4Now 4y agoThe background image doesn't render on the homepage in Firefox. This makes the blog links appear as very light blue text on a white background. Makes me wonder how it renders for the crawler and whether it's getting flagged for invisible content.
- ducklingquack 4y agoHad the same problem due to negative SEO campaigns by naughty competitors. Wrote to Bing Webmaster Tools Support team (https://www.bing.com/webmasters/help/webmaster-support-24ab5ebf https://www.bing.com/webmasters/help/webmaster-support-24ab5...) and after a lengthy process got a response that ”the issue” had been addressed. It’s been a few months since and my website is indeed back in the search results so I advice whoever is having this problem to reach out to Bing.
- obarthelemy 4y ago"Southern gentleman" conjures up the image of a white-suited Kevin Spacey in "Midnight in the Garden of Good and Evil". Not sure it makes for a strong claim to respectability/morality, both fictionally and IRL ;-p
- mtlynch 4y ago>One “out there” reason I can think is that I use Amazon Affiliate links on my Bookshelf and my /Uses page and that triggers a shadow ban? It's probably not the reason, but it's worth noting that the author is using Amazon affiliate links in violation of Amazon and FTC rules because they're not disclosing the fact that they profit from purchases through their links. Per Amazon: >Anytime you share an affiliate link, it's important to disclose that to your audience... you must (1) include a legally compliant disclosure with your links and (2) identify yourself on your Site as an Amazon Associate with the language required by the Operating Agreement. https://affiliate-program.amazon.com/help/node/topic/GHQNZAU6669EZS98 https://affiliate-program.amazon.com/help/node/topic/GHQNZAU... Per FTC: >As for where to place a disclosure, the guiding principle is that it has to be clear and conspicuous... Consumers should be able to notice the disclosure easily. They shouldn’t have to hunt for it. https://www.ftc.gov/business-guidance/resources/ftcs-endorsement-guides-what-people-are-asking#affiliateornetwork https://www.ftc.gov/business-guidance/resources/ftcs-endorse...
- vcg3rd 4y agoYour site doesn't get a hit, but it's referenced enough to find it. One hit was https://www.seoaudit365.com/domain/daverupert.com https://www.seoaudit365.com/domain/daverupert.com It says your IP doesn't direct to your site. I wonder if that's the problem.
- tambre 4y agoIt would be quite surprising if reverse records were the reason. The vast majority of sites certainly don't have them pointing back. Impossible to do with many providers and probably basically all CDNs.
- Macha 4y agoIt's not a missing meta description at least, the same is true for my site and I'm still ranking relatively highly for some keywords on DDG.
- crazygringo 4y agoI may have found the answer, and I've seen this before (it happened to me once). It's when a different spam site copies your content wholesale, and a search engine decides they're the "original" site, and you're the spammy copycat. Because if you put the headlines (in quotes) from two of his recent articles into Bring, e.g. either "Megan Smith explaining the General Magic prototyping process" or "Denialists, Alarmists, and Doomists", both point as their first result to a URL starting with "https://www.scien.cx https://www.scien.cx" which seems to be the spam site with a copy of each article. (The URL isn't loading right now, however, when I try to visit.) How to fix it really depends on what techniques they're using to mirror your site, of which there are many. Example search and resulting URL: https://www.bing.com/search?q=%22Megan+Smith+explaining+the+General+Magic+prototyping+process%22 https://www.bing.com/search?q=%22Megan+Smith+explaining+the+... https://www.scien.cx/2022/12/25/megan-smith-explaining-the-general-magic-prototyping-process/ https://www.scien.cx/2022/12/25/megan-smith-explaining-the-g... Compare with Google getting it right: https://www.google.com/search?q=%22Megan+Smith+explaining+the+General+Magic+prototyping+process%22 https://www.google.com/search?q=%22Megan+Smith+explaining+th... https://daverupert.com/2022/12/megan-smith-general-magic-prototypes/ https://daverupert.com/2022/12/megan-smith-general-magic-pro...
- TrueGeek 4y agoWhat should you do when another site copies your content like this then?
- gkbrk 4y agoYou send a DMCA.
- pungentcomment 4y agoChances are that the site will be hosted outside the US and the DMCA is a US only copyright law afaik. I don't think it's applicable outside the US.
- 4y ago
- Hani1337 4y agoduckduckgo is a joke. use gibiru
- shirback 4y agoMicrosoft thinks you're Dave Rubin because.. well, it's Microsoft.
- svnpenn 4y agonumbers look like garbage, I think this is the culprit: font-family: system-ui, 'Noto Emoji', sans-serif;
- beej71 4y agoI have exactly this problem. Beej's Guide to Network Programming is indexed just fine. Beej's Guide to C won't index. The automated tool says it's in violation of some unnamed rule, but I can't figure out which. There's zero SEO, tracking, or ads, and the content is educational and G-rated. All the other guides index just fine. I asked for a review and they came back with the same ambiguous message. Eventually I just gave up. Recently I split the C guide in two. I'll have to check to see if that made any difference. But it left a bad taste, and now I don't trust Bing or DDG to provide complete results. Google's overrun with spam, but at least my stuff actually shows up on Startpage.
- KRAKRISMOTT 4y agoIt is interesting since Beej's guide is probably the most famous C/Unix programming tutorial after K&R.
- abdullah2993 4y agoWhats K&R?
- jinwoo68 4y agoKernighan & Ritchie: https://www.amazon.com/Programming-Language-2nd-Brian-Kernighan/dp/0131103628 https://www.amazon.com/Programming-Language-2nd-Brian-Kernig...
- sigio 4y agohttp://cslabcms.nju.edu.cn/problem_solving/images/c/cc/The_C_Programming_Language_%282nd_Edition_Ritchie_Kernighan%29.pdf http://cslabcms.nju.edu.cn/problem_solving/images/c/cc/The_C...
- beej71 4y agoI don't know if I'd go _that_ far. There are a lot of great books out there. But I'm pretty sure mine is the greatest that you can't pay money for. ;)
- dazc 4y agoSites get de-indexed from DDG and Bing all the time because negative SEO attacks (intended for your Google traffic) have a devastating effect. Why, Bing does not totally ignore bad links. The good news, all sites I've seen affected by this recover after a few weeks.
- _8j50 4y agoHN taught me first hand on how horrible shadow banning is. You all tolerate this here so it's mighty hypocritical of you to criticize Bing. I'll say it again: It comes down to how you treat people. Treat others the way you want to be treated. No one wants to be shadowbanned and we can all agree it is a decidedly cowardly and cruel thing to do. And you can't use "quality" or anything short of being coerced as an excuse. Techniques and technologies to moderate people without shadowmodding at scale are mot just there but very well established. A site for technologists has no excuse to shadowmod other than elitism amorality.
- _8j50 4y agoLook at my post history and karma lol. I get exactly 3 karma and it gets hidden. I should do a line chart and post it somewhere outside HN lol.
- maxbond 4y agoHN's "shadow bans" aren't hidden and allow you to keep posting. Isn't that more tolerant than most other forms of banning? I have showdead turned on. I see fresh accounts that are automatically dead on each comment which are legit contributions, probably because they're using Tor or a widely abused VPN; that's the only common miscarriage of HN moderation I regularly see, and I vouch for house comments. Those accounts should be in the clear after a week or something like that. I have a couple comments I feel shouldn't be dead, but I can see how others would feel differently, and I have I believe 3 dead comments out of >3000 (many of which expressed views others vocally disagreed with, and I generally feel my views are not particularly popular on HN). But most of the dead comments I see are obviously harmful to discussion. The last time I saw hate speech from a banned HN account - was earlier today. What is it I'm missing here? It's all well and good to say, treat others as you'd like to be treated. But I don't want to be harassed either. So I forgo harassing people sure. But what's to be done about the people harassing me? Are you perhaps unaware of a phenomenon called the paradox of tolerance where, if you extend universal tolerance to everyone, including those who use their speech to silence others (through threats, harassment, shouting over people, poisoning the well, etc), you still end up with a forum in which not everyone can share their ideas?
- imwillofficial 4y agoThis article was a waste of time. Nothing was learned of value.
- Twirrim 4y agoOkay, so this prompted me to look at Big Webmaster tools. It looks like they recognise my sitemap file, but haven't bothered to index it in nearly a decade. "Last processed: 6/6/2014". I know I don't post new content that frequently but... WTF?
- cramjabsyn 4y agoIt’s not a shadow ban if you can easily see that you’ve been banned. The point of a shadow ban is for the banned user to not notice. This sounds like a regular old ban/blocklist.
- kordlessagain 4y agoNo mention of robots.txt on the post, nor here in the comments. After a bit of poking around, it would appear Bing may need an allow block to crawl. I don't know what DDG does, but the author's site effectively has nothing in the robots.txt file, other than a commented out Disallow block. From doing this before in the past, I suggest include the following: User-agent: * Allow: /
- RandomWorker 4y agohttps://daverupert.com/robots.txt https://daverupert.com/robots.txt Maybe change this? Simply add: User-agent: * Disallow: To allow all crawlers to the site.
- deleted 4y ago[deleted]
- jasmer 4y agoI wonder if we need basic regs given that search is a public good, including best practices such as SE transparency. If they search you, they have to say something about the parameters and the results etc..