14 ms·
Basecamp was under network attack
- rdudek 13y agoIs it just me or are these attacks becomming more and more common? I hope we can get some more details on the attack like the origination of it, type used, and what steps were take to mitigate it. I always use information like this as a learning opportunity :)
- rickyc091 13y agoGithub provided some information on the attacks being performed. https://github.com/blog/1796-denial-of-service-attacks https://github.com/blog/1796-denial-of-service-attacks
- alandarev 13y agoWhen even the governments use DDoS [1] as a method to 'turn-off' services they don't like, it will be a very long path to fight. [1] - https://www.quakenet.org/articles/102-press-release-irc-networks-under-systematic-attack-from-governments https://www.quakenet.org/articles/102-press-release-irc-netw...
- vidar 13y agoWould CloudFlare help here?
- genwin 13y agoI've been wondering myself, if CloudFlare helps against DDoS attacks when the page is dynamically generated for each user. For static pages it should help.
- joshmcmillan 13y agoIf the attack is working by essentially flooding Basecamp's network links until they reach capacity, then yes, it could. CloudFlare could simply filter out malicious traffic and only pass on legit requests to Basecamp. That's obviously very much dependent on the kind of attack and whether CloudFlare has more network capacity than Basecamp (which I would imagine is highly likely).
- singlow 13y agoCloudFlare does more than just caching. Even on non-cached pages it can filter and otherwise mitigate traffic that it has identified as malicious.
- kyrra 13y agoProbably. Mitigating a DDOS (from my understanding) has two important things that need to happen. (1) You need a larger incoming pipe than the data being sent to you. (2) you need to ignore invalid requests so you don't flood your outgoing pipe as well. Properly ignoring invalid requests can be a challenge, the process of doing so will depend on the type of attack being used. SYN floods can difficult since the src IP is most likely invalid. The attacks we've seen with DNS and NTP amplification are difficult as the attack isn't trying to get your servers to respond, they are just flooding your incoming pipe with data. If they are trying to abuse some page within your application you can more easily mitigate it as you'll know the source IP of the request so it can be blacklisted. EDIT: a few more details: SYN flood: http://en.wikipedia.org/wiki/SYN_flood http://en.wikipedia.org/wiki/SYN_flood DNS Amplification: http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack http://blog.cloudflare.com/deep-inside-a-dns-amplification-d... As for mitigation, while we hear about Cloudflare a lot, AT&T and other large providers can provide DDOS protection for leased lines[0]. Basically what happens, before the data gets to your leased lines, traffic headed to you will go through AT&T's DDOS detection/prevention systems that attempts to filter bad traffic. This type of service would apply more to companies like Linode or possibly the datacenter that they are housed in. [0] http://www.business.att.com/enterprise/Service/network-security/threat-vulnerability-management/ddos-protection/ http://www.business.att.com/enterprise/Service/network-secur...
- illuminated 13y agoDepends on the scale/power of attack. The latest hits (happening in the last few months) have been very large and I doubt CloudFlare would be able to successfully defend any of those while maintaining all of the current clients online. I have a client that occasionally gets this kind of blackmailing followed by attacks and they told me they use a US based company specialized in DDoS defending - until now the defense was pretty efficient. I've never bothered to ask for a name, but I guess it's a known one in the "network industry".
- eastdakota 13y agoYou're mistaken. CloudFlare has mitigated many the largest DDoS attacks in history, including some that have exceeded 400Gbps. These recent extortion-based attacks are large, but they are typically 1/10th the volume of the largest attacks we see. For instance, Meetup has publicly stated that they used our network to stop a similar attack. Many of the other recent victims have used CloudFlare as well. Because of the unique design of our network, I'm unaware of any other service that has as much capacity that can be utilized in aggregate to mitigate large-scale attacks. Matthew Prince Co-founder & CEO, CloudFlare
- larrys 13y ago"CloudFlare help" I'll leave it to others to answer this (for this situation) but keep in mind also that adding cloudflare also adds an additional layer that can fail for different reasons. That tradeoff may well be worth it for certain high visibility web properties but maybe not if you are a low value target. There are pros and cons to any decision you make that depend on specific circumstances.
- timdorr 13y agoIt depends if this attack is on basecamp.com or the IPs that basecamp.com resolves to. It appears Basecamp only has a /23, so even if they redirected traffic through Cloudflare, the attacker could still find their direct servers fairly easily and attack that IP. It's still possible to block, but not quite as easy as setting up Cloudflare.
- chimeracoder 13y ago> so even if they redirected traffic through Cloudflare, the attacker could still find their direct servers fairly easily and attack that IP. Why would it be easier for the attacker to find their direct servers if they only have a /23 - doesn't Cloudflare obscure the identity/location/IP of the server on the other side?
- timdorr 13y agoIt's only 512 addresses, so the attacker can just switch between different IPs until service degrades and keep on that address. Also, it's likely their rack/cage has a limited amount of bandwidth compared to the whole datacenter, so they can just send traffic to that range and overload the switch.
- jessaustin 13y ago...the attacker could still find their direct servers fairly easily and attack that IP. Can the upstream to the actual server restrict traffic to known Cloudflare blocks?
- scottbruin 13y agoWe've had issues with saturated upstreams and then been negotiating new ISP connections. All the ISPs I've asked (Level3, NLayer, Cogent) won't put an active restriction to only CDN blocks upstream. The ISPs will help during a DDOS but response times are slow and we haven't tried getting them to put this type of block in place yet.
- moollaza 13y agoAfter taking a look at CloudFlare's knowledge base, it seems that their services would definitely help if you were under attack. According to CloudFlare, they offer basic DDoS Protection with their plans, and it seems like you can upgrade to a business account during attacks for improved protection/mitigation. They also claim that they don't have a cap on the size of attacks they can handle. Relevant links: https://support.cloudflare.com/hc/en-us/articles/200172676-Can-CloudFlare-protect-me-against-DDoS-attacks- https://support.cloudflare.com/hc/en-us/articles/200172676-C... https://support.cloudflare.com/hc/en-us/articles/200170216-How-large-of-a-DDoS-attack-can-CloudFlare-handle- https://support.cloudflare.com/hc/en-us/articles/200170216-H... https://support.cloudflare.com/hc/en-us/articles/200170196-I-am-under-DDoS-attack-what-do-I-do- https://support.cloudflare.com/hc/en-us/articles/200170196-I...
- janlukacs 13y agoAlthough a smaller service, we were in a similar situation a couple of years ago. We assumed it was a competitor because there were not monetary requests, just a massive DDoS via torrents that lasted almost a week. Data center didn't help us in any way... it was crazy. Worst thing is that 90% of customers have no clue what a DDoS is and how hard it is to handle.
- alandarev 13y agoHow is torrents protocol used to DDoS you? I never came across torrents being used as a DDoS. I would appreciate more details on what sort of torrent attack it was, and whether you found any ways of partially neglecting damage.
- Danieru 13y agoA malicious tracker, or a peer if using DHT, can claim an IP, the victim, is active in the swarm and has valuable bits of the torrent. Then torrent clients will try to connect to the victim. The attack is pretty clever, being indirect it is hard to trace and because bittorrent allows arbitrary ports you can hit a specific ip & port pair. The one downside is the victims can be sure it is a bittorrent DDOS by checking the attacking connection's requests. The attacker's packets will contain bittorrent's magic connection bits.
- toomuchtodo 13y agoI'm always amazed at the clever ways people come up with to use non-aware clients for malicious purposes.
- jessaustin 13y agoThe attacker's packets will contain bittorrent's magic connection bits. ISTM that once you've determined bittorrent is the attack vector, the hard part is done? Is dropping by "magic bits" harder than dropping by ip/port?
- 13y ago
- swanson 13y agoSome great language there: framing it as an attack by criminals (gains sympathy from users), explains in plain-terms what a DDOS is (front door analogy), emphasizes (twice!) that user data is safe, apologizes for the likely downtime, informs people where to get updates. Probably worth bookmarking this for when you [hopefully never] have to deal with this same situation.
- deleted 13y ago[deleted]
- geetee 13y agoExplain please.
- valarauca1 13y agoThe problem he may have is its a bit to reassuring. They claim user data is safe while being under attack. This is conceptually very similar to teaching kids that if you duck and cover during an ICBM strike you'll be fine. >This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way. If this is truly 110% true, they couldn't even ssh into their servers (in before, "You don't ssh each into individual shards"). Which I'm betting they can, which means their are still attack vectors to exploit.
- kevinchen 13y agoNot sure that your missile analogy holds. Most DDoS attacks do not attempt to crack logins to servers, but rather try to flood the servers with as much garbage as possible. Besides, even if they were trying to crack the SSH password, a properly secured server (long passwords/public key auth + fail2ban) should be fine.
- valarauca1 13y agoOkay here is a better one. Just because people are blocking each other trying to run into your front door doesn't mean they (or somebody else) aren't cutting open your windows, picking the lock on your garage door, or trying to climb down your chimney.
- robgering 13y agoHow do larger companies (like Basecamp) prepare for these kinds of risks? Do they contract with DDoS mitigation firms beforehand, or do most tend to hire help only when they are actually attacked?
- lawncheer 13y agoDDOS firms (prolexic etc) are really expensive, I would imagine they do it on an as-needed basis. From my experience working at a datacenter, the first line of defense are the techs in the datacenter, for most attacks, they can blackhole offending IPs etc, and mitigate it. When it gets to the point of being something huge though, like the meetup.com attack, I would imagine they call in an outside firm.
- mobiplayer 13y agoSurprisingly, they usually don't.
- wehadfun 13y agoWhat law enforcement do you call in these situations. I imagine it would be a waste to call local police. I don't know how you would get feds to pay attention?
- codazoda 13y agoAssuming the ransom request wasn't fake. It's pretty likely that the attack came from outside the US. Law enforcement will probably not be able to help at all.
- revscat 13y agoKim Dotcom would like a word.
- Xylakant 13y agoWhy not? The US. Law enforcement obviously doesn't have jurisdiction, but as long as a DOS is illegal in the country that the attacker sits in, the US Law enforcement should investigate and hand off to a partner agency in that country, acting as liaison and serving a request for extradition. It's a different matter if the attacker is based in a country where DOS are legal or that doesn't have any extradition treaty with the US, but that still needs to be established.
- citruspi 13y agoI believe that the Federal Bureau of Intelligence investigates and prosecutes cyber crimes[0]. [0]: http://www.fbi.gov/sanfrancisco/press-releases/2011/charges-in-distributed-denial-of-service-attack-against-santa-cruz-county-website http://www.fbi.gov/sanfrancisco/press-releases/2011/charges-...
- philtar 13y agoInvestigation, not intelligence. CIA does intelligence. Or you could've just typed FBI, like a normal person.
- joevandyk 13y agoHas anyone defended a DDoS attack on an application hosted on Amazon's AWS/EC2? If so, how did that go? Did Amazon help?
- mgorsuch 13y agoI was involved with a company that received several attacks on AWS. We were premium support customers, and were able to work with our AWS TAM to get a mitigation device in place and turned on. It was a bit shaky at that time, as this was not a common service offering. Things may be better now.
- barkingcat 13y agothey did get a blackmail email so it does seem like they are being targeted by someone.
- codelittle 13y agoWhoever is doing this thank you for reminding me how important Basecamp is to my business. I hope they hunt you down.
- rootuid 13y agoA perfect time for those affected to test drive BaseCamp's competitor https://www.teamwork.com/ https://www.teamwork.com/
- xxdesmus 13y agoclassy.
- CanSpice 13y agoDoes anybody know how many companies, upon receiving a blackmail "give us $300 or you'll be DDoSed" email, pay it? For every meetup.com or Basecamp that resist, how many actually give in to the blackmailer's demands?
- cmdkeen 13y agoIt isn't $300, it's "up to $50,000"[0] I've seen articles before saying online gambling websites often do pay up as the downtime isn't just lost revenue but customers going elsewhere. [0] http://www.prweb.com/releases/2012/4/prweb9455636.htm http://www.prweb.com/releases/2012/4/prweb9455636.htm
- TacticalCoder 13y agoI take it at one point people will start to believe that I work for OVH (I really don't) but... OVH has a mandatory DDoS protection on all its dedicated servers: fees have been slightly raised to take that mandatory protection into account. There are a few gotchas, including if I understand it correctly the need to "retry twice" when you try to SSH in your server when a DDoS is going on but... OVH doesn't even feel a 85 Gbps attack (let alone a 20 Gbps one like in the article). They can deal with attack much larger than that automatically. They seem to have very good DDoS protection against the "flood" type of DDoS. And this is pretty much transparent to users. I hope more and more hosting company start implementing similar anti-DDoS features: more competition would bring better protection against flood-type DDoS and cheaper price. Here's the explanation as to how their system works (in french but there are several graphics): http://www.ovh.com/fr/a1164.protection-anti-ddos-service-standard http://www.ovh.com/fr/a1164.protection-anti-ddos-service-sta... Basically as soon as a DDoS trying to saturate your server(s) is detected the attacker faces the problem of needing to DDoS... OVH itself. And the DDoS doesn't even make it to your server while the legitimate trafic still does. I find it great that there are people actually looking for solutions to the DDoS issue.
- cordite 13y agoI have a service on OVH myself. Though a friend at another related service had been kicked from two VPS providers due to receiving a few DDoS attacks. These providers claimed it was against their Terms of Service and ejected him as a customer. That day he learned it is best to keep offsite-cross-company backups of everything, since he did not get a single byte from his machines.
- olsonea 13y agoI wonder if there will be a day where on-premise solutions will be touted as the solution to the DDoS vulnerability of cloud-based solutions, in much the same way that there seems to be an ebb and flow between fat and thin clients over the course of computing history.
- samplonius 13y agoBecause on-premise solutions are even more vulnerable to DDoS. A large data centre will have large amounts of connectivity, giving you a lot of head room for most types of attacks. But in this case 20Gbps of extra traffic was too much too. What on-premise solution can handle 20Gbps of extra traffic? And I don't think Basecamp is technically "cloud", but collocated. They appear to own most or all of their servers.
- Nacraile 13y agoIf you define on-premise as being accessed over a private network (which seems to be the idea here), then it is not directly vulnerable to DDoS at all, because it isn't reachable from the public internet.
- norswap 13y agoCrime, crime, crime, criminal. While technically (and probably also morally) true, was I the only one to find the emphasize weird?
- Aqua_Geek 13y agoI thought it was weird until he mentioned the blackmail. DDoS-ing for the lulz is one thing, doing it and then blackmailing the victim to get it to stop is a whole other level.
- akassover 13y agoWe got hit by a DDoS about a year ago. Rackspace (who normally has amazing support) quietly null routed us and went about their day. No heads-up, trouble ticket, or any other form of notification. They didn't even put a note in our account so when we contacted their support to figure out why our servers were unresponsive outside their network the poor guy who answered the phone was just as confused as I was. We've taken some steps since then to hopefully reduce our vulnerability. I'd be really interested in a DDoS protection best practices guide for small SaaS businesses.
- akassover 13y agoIt's worth adding that when we got hit, we were relying on "security through obscurity". I slept well at night because I thought nobody would be interested in DDoS'ing little ol' us when there are plenty of big fish out there to go after. I'm sure a few of you out there are readying this thinking "too bad for Basecamp, but this will never happen to us because we aren't an interesting target." That's what we thought too...
- alexcroox 13y agoYep Rackspace did little to nothing to help us but null routing.
- gk1 13y agoI'm running a small SaaS business. I'm curious to hear what steps you took to reduce your vulnerability. Could you please share so others can take the same steps?
- akassover 13y agoThe biggest thing we did was remove our dependency to a single IP (this was a unique requirement of our business). We also improved our firewall and upped our managed service level. We're not 100% bullet proof now, but definitely better than we were. I'd be happy to go into more detail offline.
- gk1 13y ago
- coreymgilmore 13y agoSomething along the lines of CloudFlare could be an option here. However, if the attacker does indeed know the actual IP of the Bootcamp servers (and Bootcamp allows traffic from IPs other than CF) that point is moot. Set up CF, only allow traffic from CF. On another note, having CF monitor an attack like this could help them do more research into mitigating these attacks in general and allow them to try and hunt the attacker. They tend to make things like this public which would benefit everyone.
- devicenull 13y agoI personally wouldn't do any business with cloudflare, while they're still hosting the various booter sites where you can pay to run these attacks.
- bybjorn 13y agoCloudFlare is hosting booter sites?
- devicenull 13y agoYea, see https://news.ycombinator.com/item?id=7459904 https://news.ycombinator.com/item?id=7459904
- xxdesmus 13y agoCloudFlare does not host any website or it's content actually. They are not a web hosting service.
- deleted 13y ago[deleted]
- grey-area 13y agoIf you're going to make accusations like that, you should really back it up with extensive proof.
- demoncore 13y agoI'd really like more detail on the nature of the DDoS and I'm surprised more posts here haven't asked for that. What kind of packets are being used?
- ing33k 13y agois it the first time they are facing this sorta attack ?
- quarterwave 13y agoA speculative thought: Apart from being distributed, the insidious power of DDoS appears to lie in "subscriber-calling-server". Why not go the other way around? At least only for specific subscription services, not general purpose web access. The situation of a DDoS attack is first communicated by the web service provider texting a subscriber, who texts back their present IP address. The web service provider then "calls" the subscriber from a hitherto unknown IP address. Of course, that address could be leaked too, but at least it's not obvious public knowledge like a DNS entry. Sounds like circuit switched telephony/modems rather than packet switching, but can it be implemented in software?
- sirsar 13y agoA great deal of consumers are behind NAT, and punching through that is a huge pain. UPnP is sketchy, STUN is difficult, and custom schemes like uTP are undocumented. You'll get the occasional consumer who is willing to forward a port just to connect to your service, but not very often.
- ivanca 13y agoIs there something like cloudfare but more aggressive? Like something that tries to find exploits on the machines used in the attack and try to shut them down, close their internet connection or inject a self-targeting DNS or something of the sort?
- Nacraile 13y agoIANAL, but I've seen this discussion come up multiple times, and the problem is that the counterattack would technically be illegal. The fact that somebody else has already broken the law in order to compromise an innocent bystander does not give anybody else the right to do the same thing. Vigilantism is as illegal on the internet as it is in the real world. This is a huge constraint for the people (e.g. at Microsoft) who work to identify and take down botnets: they expose themselves to significant legal/PR risk if they do anything harmful to the bots.
- ivanca 13y agoBut this could be considered self-defense which is granted by most law systems.
- Nacraile 13y agoAgain, IANAL, but my understanding is that the concept of self-defense is specific to the use of force, rather than broadly applicable. You'll find it difficult to prove an immediate thread of physical harm from a DDoS. And even if it were legal, you'd still have to deal with all of the "$SELF_DEFENDER broke my web site" PR unpleasantness from the innocent bystanders.
- PeterisP 13y agoSelf-defense is granted only for a direct, immediate physical threat - for example, if someone is blackmailing you, defrauding you or extorting "fire insurance for your warehouse" then self-defense doesn't allow you to do anything to them; if you smash the computer of a blackmailer, it's just as any other computer-smashing.
- 13y ago
- reshambabble 13y agoEvery business experiences fires that they have to put out, and their transparency on what exactly the issue is keeps us informed and on their side.
- filet 13y agoI've had really negative experience with these type of criminals. I was hired as a CEO at an <unnamed> company ($200m+ revenue) and we were hit by this type of attack. Every second of being down cost us literally $10k, so we quickly negotiated with criminals for $5k one time payment and they stopped the attack. Unfortunataly a few weeks later we were hit by 3 new attacks. Apparently the word had spread and these new attackers demanding $50k. We were not going to pay $50k but I was also unable to stop the attacks. I was let go a few days later as we had a down time of 2 days and I wasn't able to fix this problem. Crap.
- PeterisP 13y agoThat's a good reason why it's never a good idea to pay for DDOS threats - in many other popular extortion scenarios such as kidnapping, blackmail w. secret info or mafia 'protection money' for storefronts, the deal generally doesn't allow other, new attackers to make the same demands, so you actually are getting some protection - but here it does simply mark you as vulnerable.
- sergiotapia 13y agoSame goes when bribing a cop here. If you bribe too much you're targeted as easy money among the other cops here. Say for example you're caught driving without your insurance, you bribe and then every other cop knows you don't have insurance and squeeze you for money left and right. Source: 3rd world south america
- driverdan 13y agoWhy did it fall on your shoulders and not the CTO / tech team? How did they let you go in a few days? They called a board meeting immediately after to fire you?
- stock_toaster 13y agoThis is another great example of why I wish there was support for disabling commenting on gists.
- ambrop7 13y agoI'm wondering what happens to botneted subscribers from which the attacks originate. Is any attempt made to locate them and contact their ISPs? I think there should be, and subscribers found to be participating in the attack (presumably unknowingly) should be disconnected immediately. After all it's the subscribers' responsibility to keep their computers botnet free. Launching a DOS attack, even unknowingly, is probably violating the contract they signed with their ISP.
- drewblay 13y agoForget baecamp. Setup a webserver throw Colalbtive on it. Now you are in control of your data (you are now also responsible for the uptime). Colabtive: http://collabtive.o-dyn.de/ http://collabtive.o-dyn.de/
- stcredzero 13y agoWe need the kind of concerted attention paid to this stuff that we gave to horse thieves in the Old West.
- griffinheart 13y ago> When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading. I'm interested about what he means by quarantine. Does it mean that ISP's will stop accepting traffic going to their servers?
- Allower 13y agoYet another reason we should be utilizing P2P WAY more often