9 ms·
Spamhaus Nightmare: Domain Shut Down, No Notice, Over A Million Pages Down
I am writing this here on HN, because at the moment you cannot access our blog. Our domain name was shut down this morning, and I'm trying to get it back. Here's what happened...
Our company provides tools to help people put together pages for their businesses. Our free tool has been used to create over million page tabs on Facebook. Unfortunately but predictably, sometimes bad people use our app. Like spammers.
Overnight, our domain was blacklisted by Spamhaus because one of our pages contained spam. (Anybody want a free iPad?)
We run our infrastructure on Heroku, and use Bluehost for domain names. Well, as soon as Bluehost recieved notice from Spamhaus, they shut off the DNS for our domain. All million plus pages, gone in the blink of a DNS propagation.
Thankfully we were able to switch over to [appname].heroku.com for now and most of the pages are back, but we have paying customers who are in the dark because they rely on our custom domain name.
Our product, that over a million people rely on, suddenly ceased to exist. No advance notice. Nothing we could have done to stop it. Because of ONE bad apple.
This kind of thing will happen in SOPA world, if we let ourselves get there. But instead of being able to call my registrar and yell at them, I would have had to call the government, and oh-by-the-way they might fine or imprison me for having hosted spam.
Let me end with a practical, really-important-to-me-right-now question: is there any possible way to not get randomly nuked by Spamhaus?
- zackzackzack 15y agoFirst suggestion: get an ip address people can remember. Not very practical I know, but I guess that is the only way to get by without DNS at the moment. Also: you've emailed your customers the new address yes? Even if it is only temporary? Maybe buy a new domain and point them towards that:"Please use [FINGSOPA].com while we get everything back to normal."
- ianlevesque 15y agoSo its time to add Bluehost to the list of companies too unreliable to do business with.
- Nick_C 15y agoFunnily enough, I just got my Dreamhost bill for the next year and was thinking of switching to Bluehost due to cost. Not any more.
- elliottcarlson 15y ago"is there any possible way to not get randomly nuked by Spamhaus?" I guess the first step is to set up better monitoring services to prevent your system from being abused by even one bad apple. Try to catch the abuse as quick as possible so you won't raise red flags. Additionally you should possibly work on segmenting out your customers. If your paying customers are important to you, use a different system for them. If this has the possibility of happening again you don't want to hurt those customers from a similar thing happening again.
- kposehn 15y agoExcellent point about that. You could always break up free users into a different domain than paying, and even break those up further from there. Not pretty, but it would help ensure against it.
- jpadvo 15y ago> I guess the first step is to set up better monitoring services to prevent your system from being abused by even one bad apple. I'd love to, but when you have the volume we do there are going to be false negatives. Bad apples will slip through. And if somebody slips through, we're vulnerable for getting blacklisted. > If your paying customers are important to you, use a different system for them. This is a very good plan. I'm definitely looking into that...
- elliottcarlson 15y ago> I'd love to, but when you have the volume we do there are going to be false negatives. Bad apples will slip through. And if somebody slips through, we're vulnerable for getting blacklisted. This doesn't mean you have to auto-ban people - but you could easily setup listings that you can quickly glance at to see what your monitoring found. If you are picking up too many false positives, then you can refine your monitoring. Yes - you can't prevent all of them, but once your system starts getting abused you have to assume that others will do it as well. Additionally, you could very well be losing money in service costs due to these people (I don't know your business model so it's just an assumption) - you want to protect that as well.
- freejack 15y agoMy recommendation would be to run your own DNS on your own IP addresses. Even with the IP shortage, you should be able to get a small block delegated to you that you can use for your mission critical apps. Once you've got that arranged for, its a fairly trivial task to find a registrar with policies more complimentary to your business. If its mission critical for your business, then you can't afford to think like a victim. Take charge of your infrastructure where you have to. Relying on third parties is lean, but not always effective - a small amount of fat in the right areas can give you a lot of flexibility (and insurance) that you might not get when you rely on a third party.
- jpadvo 15y agoI didn't realize that I could set up my own DNS on my own IP addresses. Thank you very much for the suggestion, I'll look into this. By the way, do you know off the top of your hand any such registrars with more complimentary policies?
- freejack 15y agoIf you are in North America, check out https://www.arin.net/resources/transfer_listing/needers.html https://www.arin.net/resources/transfer_listing/needers.html for details concerning address delegations. I work for a registrar, so here are my biased recommendations - me (hover.com) or EasyDNS. But don't take my word for it - do your own homework. Its worth knowing what your risks are and no amount of free advice on HN can replace that.
- nirvana 15y agoCheck out Moniker. They have a no-nonsense policy and are in the business of protecting domain owners.
- dholowiski 15y agoCorrect me if I'm wrong, but even if you run your own DNS servers, can't your domain name registrar still decide to take away your domain name?
- RealGeek 15y agoBoycott Bluehost?
- sp332 15y agoNo need for a boycott. If they're so unreliable, customers will move away themselves.
- sycren 15y agoI'm up for renewal in february with them, could you suggest a better host?
- Karunamon 15y agoI'm a fan of Lost Signal, I've got a single dedicated server box with them right now. It's a small operation but very hands on. https://lostsignalweb.com/services/shared https://lostsignalweb.com/services/shared
- zabraxias 15y agoI use linode but I was happy with slicehost before also. Granted I've never run popular apps/websites on these so I am not sure how they'd treat abuse complaints.
- fleitz 15y agoServerBeach / Peer1. Find a hosting company that charges enough to have a real rep to talk to. Talk to that rep, explain what you're doing and get acceptance that this complies with their TOS in writing. The rep will get you this because they want commission. When someone asks you for hosting let your rep know, you'll get commission too, and your rep will love you and go to bat for you against stupid AUP violations and you'll have it in writing that your activities are acceptable.
- staktrace 15y agoI used to use bluehost until their gradually declining level of service was just too much. I switched to dreamhost and have been happy ever since. Highly recommended.
- battaile 15y agoWhile I'm sorry this happened to you, and I'm as anti-SOPA as anyone (have called my congress(wo)man, called Boehner and Canter when it looked like they were going to sneak the vote through last week), this has nothing to do with SOPA, and trying to invoke the name for something that you should've been better prepared for is kind of a discredit to the cause.
- glombus 15y agoI agree. I am anti-SOPA, but this is not censorship so much as an overzealous attempt to stop spam, executed poorly. Did they do a wildcard block/hold/whatever on a top level domain of yours *.mydomain.com so all your sub domains got blocked? I'm guessing some goober at bluehost just went one step farther than he should have in just removing the one DNS entry, and they definitely should have contacted you. Sounds like poor customer service.
- chernevik 15y agoIn the U.S., censorship will very likely travel under the guise of some "overzealous attempt" to stop this, that or the other. SOPA's censorship problem is not the explicit endorsement of censorship but the precedent of mechanisms and principles that will make censorship far easier to implement and "justify".
- fr0sty 15y agoAnd if SOPA passes you will see suspiciously simil overzealuos attempts to stop piracy. Same song, different verse but this time you get the federal government involved which is a whole new level of fun.
- glombus 15y agoI agree http://news.ycombinator.com/item?id=3409045 http://news.ycombinator.com/item?id=3409045
- FreebytesSector 15y agoThis is true, and it makes me thing of all of the private sector solutions that are already available. The market creates the solutions and does not need government interference slowing it down or making dispute resolutions more complicated.
- subway 15y agoThis doesn't sound right. Since when does Spamhaus police site content? I'm pretty sure they primarily go after folks sending out spam email, not after websites containing spammy pages.
- elliottcarlson 15y agoThey also go after places that enable spammers - such as when they blocked Google Docs [1] http://news.softpedia.com/news/Spamhaus-We-Blocked-Google-Docs-Not-Gmail-153093.shtml http://news.softpedia.com/news/Spamhaus-We-Blocked-Google-Do...
- whortleberry 15y agoThey added Google Doc ip addresses to their RBL, so that SMTP traffic from those IPs would be blocked by those who chose to run Spamhaus' blacklist.
- Vivtek 15y agoThey do list spamvertised domains. It is incredibly irresponsible to yank a domain on that basis alone. Bluehost is at fault here.
- brador 15y agoIf my host did that then reversed the decision I would still be moving out of there as fast as possible. There's absolutely no reason to be giving second chances to online services with so much competition about, on what is, essentially, a commodity.
- dbe 15y agoHave you tried using Spamhaus's Blocklist Removal? http://www.spamhaus.org/lookup.lasso http://www.spamhaus.org/lookup.lasso
- dholowiski 15y ago"is there any possible way to not get randomly nuked by Spamhaus?" As any email administrator will tell you, "no". The best you can do is take measures to prevent abuse coming from your domain name/IP, but bad things still do happen. You are still at the mercy of spamhaus (and other rbl providers).
- kfcm 15y agoOf course, you could just go to Spamhaus itself and attempt to remove your domain from the DBL: http://www.spamhaus.org/lookup.lasso?dnsbl=domain http://www.spamhaus.org/lookup.lasso?dnsbl=domain It could be your registrar is just running an automated process based upon that.
- jimbobimbo 15y agoUnfortunately there's no guarantee that anyone would escape Spamhaus' "love" - they and other RBLs do more damage than spammers, in my opinion. The real WTF in this story is Bluehost's reaction: shutting down DNS on one notice from Spamhaus, really!?
- whortleberry 15y agoMore damage than spammers? I see you've never run a mailserver. I have and I know that Spamhaus are one of the good guys doing a hard thankless job, risking lawsuits and threats, in order to keep email as a useful tool. Spamhaus' RBL is the most reputable of all of them, thanks to years of hard work and sacrifice. The only people who don't like Spamhaus, in my view, are those ISPs who were happy to make money from selling connectivity to spammers while pretending in public that they hated spam. Them, and people who don't understand what Spamhaus do, like the author of this article, and who think Spamhaus are to blame for their troubles.
- SageRaven 15y agoThe GP is correct. As one who has worked in the trenches as a mail admin (small potatoes, granted: a few small clients and a couple of small hosting companies), my observation has been than customers bitch way more about the MX servers which reject mail from our servers than the amount of spam in their own in-boxes. They don't give a shit that the recipient is rejecting legitimate mail -- they blame us for their problems. All because some asshat with a copy of TheBat! signed up and managed to send out a couple hundred "Russian bride" spams before we were alerted and nuked the account. I could probably fund a semester of college for some random kid with the time I've been paid to waste on de-listing and convincing idiot admins that one of their customers really wants to get mail from one of mine. Sure, 99.9% of email hitting the typical in-bound relay is spam, but CPU, RAM, and disk I/O are cheap. Do per-inbox statistical filtering and let the user decide what spam is. Better yet, let client-side filters do the work. Do you think any person would stand to allow a US Postal carrier decide what was junk mail and then not deliver it? People just need to buck up and put in a little of their own effort. I haven't used an RBL (even if its just one in a battery of weighted tests, such as with Spam Assassin) due to my loathing for the vigilante nature of the RBL scene as a whole. If you operate an RBL -- fuck you. If you are an admin that rejects mail based solely on being listed in RBLs, then fuck you, too. I know I sound like an asshole myself here, but the existence of RBLs has caused me and various mail end-users way more pain than any spammer has. Bitter? Nah. As a mail admin, I want to throw SMTP out the window. It wasn't spammers that killed the protocol, but rather the growth of use of RBLs. Rant aside, I do have a question to contribute to the discussion: Has one of the larger RBLs ever listed one of the huge mail providers (Gmail, MSN/Hotmail, Yahoo?) for any length of time? I know I've gotten spams and scams from all three.
- conductor 15y ago"is there any possible way to not get randomly nuked by Spamhaus?" Get the list (like the level1) of "evil" corporations/governments ip ranges and show a picture of a pink elephant to them instead of your real content.
- yaix 15y ago>Bluehost I'd sue them for damages. WTF do they delete your domain from their name server?! Get a more reliable registrar/name server. Spamhaus or similar black listers can always accidentially list you. Go to their site and remove your domain. No sane person/comapny should immediately assume anything but a accidential listing.
- fleitz 15y agoUse a reputable host, you may have to pay more than a few dollars a month. I've dealt with numerous spamhaus complaints, they generally result from idiotic users who send messages to spamhaus instead of clicking unsubscribe. If you spend an hour creating a really detailed form letter response it makes the AUP tickets go away quickly. When I sign up for hosting I detail exactly what we do and pay appropriately, most 'cheap' hosting places exist solely to pick up the remainder of the months service fee from a dubious spam complaint. If you spend $100 - $200 per month it's pretty easy to find a hosting provider that will let you run a single opt-in list, especially if you detail this up front in writing and refer to this in your response to any spam complaints. Web marketing shouldn't be a problem for any real hosting provider, unless you have extremely dynamic load I'm not sure why you'd bother with heroku. It's only a couple hours work to setup your own infrastructure. A quad-core server for $130 a month will run circles around what heroku provides for $130 per month.
- gasull 15y agoFor the next time, have your blog at blog.example.com, and point it to another hosting provider different than the one for example.com. That way at least you prevent the blog from going down.
- dangrossman 15y agoDoesn't work if the person pulling the site down is the registrar, and they do it by hijacking your DNS. Your pointer to the other host goes away too.
- giberti 15y agoAs a fellow Facebook tab provider (My Tab) I feel your pain. I'm concerned about how SOPA and ProtectIP will impact this class of service as it would be impossible to police all content added via tools like ours. It's already been said, but you can run your own DNS or even contract for DNS services from a wide variety of places. I would move your name and SSL certificates to a trusted registrar ASAP. Glad you were at least able to work around the issue by pointing directly to the app.
- jodrellblank 15y agoOvernight, our domain was blacklisted by Spamhaus. Nothing we could have done to stop it. Because of ONE bad apple. Because of major internet infrastructure run at whim by 3rd party blacklists, you mean. is there any possible way to not get randomly nuked by Spamhaus? Spamhaus and every service like them.
- pg 15y agoThe Spamhaus people are bad guys. I gradually realized that during the time I worked on spam filters. They presumably started out with good intentions, but the position they're in has corrupted them. It's true of a lot of the guys running blacklists. And more generally, of a lot of people in the position of police. You tend to become a mirror of whatever bad guys you're fighting. Your tactics have to match theirs, and pretty soon your principles start to as well. I suspect this tendency is so universal that you have to make a conscious effort to avoid it.
- marcamillion 15y agoHow is this avoided, on a societal level, in terms of law enforcement - other than making a conscious effort to avoid it?
- eatenbyagrue 15y agoBy having a system of checks and balances in place, that allows due process and impartial review. In the US we have the judicial process in place for this. As an interesting note, SOPA proposes doing away with much of this process in the name of "streamlining."
- pg 15y agoYes. And that's what puts vigilantes in a morally dangerous position; by definition they're not answerable to anyone. Or at least not answerable to anything more specific than the law and public opinion.
- deleted 15y ago[deleted]
- seanp2k2 15y agoWe deal with spamhaus. They have false positives just like every other blacklist. I'd agree that most blacklist opers are pretty removed from the realities of hosting / running a site / large community. Your registrar pulling your domain for this is ridiculous. I would switch ASAP to someone who cares more about their clients. This, however, is just a symptom of a bigger issue: DNS is fundamentally broken. We need a scalable, open-source, free alternative solution for SSL and DNS that does not rely on any central authority. Namecoin seems cool and it'd be sweet if people started using that. The other idea is to have a new "anti censorship" root zone, and mirror all COM/NET/ORG etc TLDs. We could pass around this info and in the event of mass censorship, people could migrate onto the new root servers. We're putting way too much power in the hands of ICANN / Verisign / any random registrar or host with our current system.
- jamespo 15y agoNo one besides the particularly clueless should use spamhaus and similar services as a black or white answer on whether to block, as they don't care about friendly fire and are run by neckbeards. Spamhas should be used as part of a body of evidence, like in spamassassin scores.
- snowwrestler 15y agoSo do we just invoke "SOPA" for any little hiccup now? Spam blacklists are not exactly a new issue on the Internet.
- whortleberry 15y agoI cannot ever recall seeing a more misleading and manipulative posting attempting to garner undeserved sympathy by falsely trying to associate one's case with bad legislation. This has nothing to do with SOPA, it is not remotely related to anything SOPA, and at worst, these kinds of false analogies only serve to weaken the case against the very real harm that SOPA will do. Spamhaus are not the villains here. First of all, you make the absurd complaint that Spamhaus "blacklisted" your domain. That is a lie. Spamhaus runs an SMTP blacklist of ip addresses that some other SMTP providers use, not all. There is no way for Spamhaus to blacklist anyone's domain. So what actually happened? Spamhaus detected a spammer website hosted on your company's ip addresses, and they did the responsible thing. They reported the spam website to the ISP hosting it. As for your claim that Bluehost shut off the DNS, why aren't you ringing up Bluehost to demand that they restore it? You might find that a better use of your time than making these absurd allegations and trying to win sympathy by making comparisons to SOPA where none exist.
- elliottcarlson 15y agoUnsure why you are being down voted - think you are 100% right.
- Gigablah 15y agoThe point here is the removal of due process. His DNS was shut off by Bluehost without any warning -- in this case there was actually abuse, but what if it were a false positive?
- whortleberry 15y agoSo why is he going around abusing Spamhaus, accusing them of "nuking" his domain? His DNS provider acted stupidly, so let him go and attack them. And stop making will accusations and idiotic comparisons to SOPA. And it's worth bearing in mind that all DNS providers, along with all hosting providers, are private companies who can cut you off for any reason, from non-payment of bills to the owner of the company disliking your politics. Talking of "due process" in the matter of private contracts between private individuals is misleading and wrong.
- Isofarro 15y agoSpamhaus normally collect evidence of abusive activity on their site. Look there first at the accumulated evidence. I'd have a look myself, but I don't know who you are, what domain you are using, what domain is being used to spamvertise. Perhaps you can post the spamhaus evidence file and we can take a look? Also, Spamhaus makes recommendations. Third parties use their lists to filter spam. It sounds unusual for a Spamhaus listing to result in a domain name shutdown, unless the DNS provider did that based on a listing. So this is not really Spamhaus' mistake (if indeed their evidence listing shows a history of hosting spamvertised websites - then there is no mistake on the listing. You could be listed either because your site/host/network has a solid history of not dealing with spam/abuse reports quickly, or because a big spam operator has landed on using your services. Are you sure it was just one site (and just advertising a free ipad)?) Yes, I understand you run a facebook static html tab content site. But that isn't a million miles away from bog standard cheap/free hosting solutions that form the bulk of spamvertised websites. Might be worth investing some time looking at the parallels and how good cheap webhosts approach dealing with spamvertised websites and spammers. So I'd suggest finding the evidence file, dealing with the problem(s) listed, then contacting Spamhaus with details of what you've done, and what's in place to reduce future abusive activity (if it's more than one site offering a free ipad). Then do something about your web hosting solution - that seems like a very weak link - either build up a better relationship with them, or move.
- bmnbug 15y agoYou mean this crap being shot all over folks comments? https://www.facebook.com/FreeiPAdd2 https://www.facebook.com/FreeiPAdd2 action="https://statichtmlapp.heroku.com/tab/1/show https://statichtmlapp.heroku.com/tab/1/show method="post" value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19'
- blhack 15y agoWould you mid deleting (or editing) this comment? You're breaking the page.
- bmnbug 15y agowww. facebook. com/ FreeiPAdd2 Like that spam site being plastered all over the place?
- bmnbug 15y agoIs this your problem? Seems you do have a spammer plastering crap on facebook. <body> <form name="redirect_form" action="https://statichtmlapp.heroku.com/tab/1/show https://statichtmlapp.heroku.com/tab/1/show method="post"> <input type='hidden' name='signed_request' value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19' ></input> </form>
- xsmasher 15y agoCan we get your app name, and a link to the spamhaus listing? It'd be nice to hear the other side of the story.