28 ms·
“We have been experiencing a catastrophic DDoS attack”
- diegorbaquero 10y ago3½ hours already. Distributed from and to many IP addresses. Seems like an attack on them rather than on specific users. :/
- tempestn 10y agoOr an attack on them as a vector to attack a specific user or users.
- ryanlol 10y ago>Update - We have been experiencing a catastrophic DDoS attack which is being spread across hundreds of different IP addresses in rapid succession, making mitigation extremely difficult. We are currently working with our upstreams to implement more complete mitigation. That's pretty harsh.
- andryou 10y agoAgreed, I hope they're able to get back online soon.
- encoderer 10y agoI've done some googling before asking here: Can anybody explain why Linode is so often targeted like this? We moved Cronitor off Linode in spring 2015. During the christmas holiday when they suffered a 2 week DDOS I thought of the family time I'd be missing that year as we did a crash migration to AWS had we not migrated when we did. I have to imagine this has been horrible for their business. I would use Linode if I needed to lease computational power, because it is still a great value vs AWS, but I could not run a high availability service there. It would feel like professional malpractice at this point.
- deleted 10y ago[deleted]
- nxzero 10y agoAsk HN: Who is behind the Linode DDOS attacks? https://news.ycombinator.com/item?id=10797795 https://news.ycombinator.com/item?id=10797795
- ryanlol 10y agohttps://news.ycombinator.com/item?id=10797827 https://news.ycombinator.com/item?id=10797827 Found the culprit.
- mappu 10y agoWe stuck with Linode after that, but the important parts now failover to Vultr.
- reefoctopus 10y agoHow did you set it up to failover? Our MySQL database is very large, and I'm not sure of the best way to ensure it is up to date on another host.
- anthony_franco 10y agoFor our setup we just have master-master replication across two different hosting providers.
- leesalminen 10y agoI've been wondering about this lately. Is it really feasible for a small (one man?) team to keep master-master MySQL replication over WAN running smoothly?
- donut2d 10y ago
- VonGuard 10y agoOK, who hosts at Linode and is very popular/pisses people off? 4chan? 9gag? Reddit? Something Awful?
- ryanlol 10y agoI think 9gag used to be on linode? I'm pretty sure none of the rest are though.
- deleted 10y ago[deleted]
- discr3t3 10y agoYeah Reddit is on AWS[1] and 4chan appears to be self-hosted (or at least it was for most of its existence)[2] [1] http://highscalability.com/blog/2013/8/26/reddit-lessons-learned-from-mistakes-made-scaling-to-1-billi.html http://highscalability.com/blog/2013/8/26/reddit-lessons-lea... [2] https://www.4chan.org/news?all#106 https://www.4chan.org/news?all#106
- busterarm 10y agoWPEngine & their customers.
- morecoffee 10y agoIt's not hard to compare this to brush fires. They happen periodically, and only the big trees tend to survive them. Linode is getting pretty unlucky here, but I would imagine that all the small time (and even the medium sized) hosting provides are going to succumb eventually. Is the end game just going to be Google vs. Amazon?
- wowaname 10y agoI'd love to see a network infrastructure and transport protocol that's more resistant to many (D)DoS attacks, because it seems like things will only worsen if it never becomes more difficult for people to attack others' servers online.
- apapli 10y agoGood luck. A DDoS is basically lots of traffic. Perhaps run IPX? (Joking)
- wowaname 10y agoIf application- and transmission-protocol-level DoS vectors are fixed, then you're left with just the raw "lots of traffic" volumetric attacks, which means your attacker has to have a lot of compromised hosts (or the right compromised hosts with lots of bandwidth). I'd say that's a reality that would be easier to handle, because you raised the bar from anyone who can develop or use a script and deploy to a few low-power systems, exploiting protocol shortcomings, to only those who have a bunch of higher-powered systems. The smaller hosting companies may still very well go out of the game if the problem worsens, even if most DoS venues do end up being mitigated. I don't know how I would respond to that as of this moment, but hopefully it doesn't have to come to that. It's already tough to find a decent hosting company in my experience.
- blfr 10y agoThere are other companies which manage to deal with DDoS attacks (OVH, CloudFlare) but you're right that none of them can be described as small.
- i_feel_great 10y agoWell, I had Linode shortlisted for an upcoming project. I hate to take them off the list because it is not their fault, but I don't want this kind of unreliability.
- tomschlick 10y agoAtlanta is their smallest datacenter IIRC with their lowest bandwidth capacity. I would still recommend them as we have had great luck with their Dallas DC. It seems to be the most resilient of them all in terms of random network outages and also DDOS attacks. However the one in December did take them offline for a few hours but after which they apparently implemented more DDOS protection. Keep in mind this could happen to anyone (Digital Ocean, Vultr, etc) and mostly their mitigation techniques seem to be to kill your VM until the attack is over.
- reefoctopus 10y ago>the one in December did take them offline for a few hours That is a bit of an understatement. Our servers were down every couple of hours for a week, and the attack bounced around to pretty much all of their datacenters.
- tomschlick 10y agoSorry I should have clarified. Our servers were down for a total of 4-5 hours spread across a week or so. After the first incident it only took a few minutes for them to come online each time, although I hear others had varying downtimes while they mitigated that DDOS. Thankfully it was over Christmas time so our clients were mostly offline.
- poopythecat 10y agoHow do you know atlanta is their smallest datacenter? Is there a list somewhere that shows what the capacity of each of their datacenters is? Atlanta got trashed the worst back in December but they beefed it up a lot. But how much? Is Dallas still bigger? Im considering migrating everything to dallas.
- finid 10y agoThis can't be good for (Linode's) business. Vultr had problems a few weeks ago, but I don't think it was DDOS-related. Somehow things have been quiet on the DigitalOcean's end.
- thezach 10y agoI used to be on Linode... then one of their techs tripped on a power cord causing a significant outage in a data center... and no I'm not joking.
- reefoctopus 10y agoThis wasn't something recent was it? Which datacenter?
- jp_sc 10y agoYou are confusing them with Dreamhost: http://www.manton.org/2011/03/brent_on_baked.html http://www.manton.org/2011/03/brent_on_baked.html
- ausjke 10y agowith linode for 10+ years and never heard about this, can you back this up with some evidence?
- Scarbutt 10y agoWhat kind of DDOS attack is the most likely happening here, a simple L3 spoofed ICMP flood?
- jonahx 10y agoCan anyone recommend a good article that explains how attacks like these work, and what is required to stop them? Also, we're on Heroku and they advertise Ddos mitigation as a feature, but "mitigation" sounds non-commital and I'm curious how they'd fare against a similar attack?
- deleted 10y ago[deleted]
- whorleater 10y agoCisco has the easiest to grasp fundamentals [1], but to really understand DDOS attacks you'll have to dig down into learning about how TCP works and network layers. Mitigation sounds non-commital because it is. I don't personally have any experience with mitigating a DDOS attack on Heroku, so I'm not qualified to talk about their prepardness, but DDOS mitigation varies quite a bit [2][3], running the entire gamut of "block a single IP address" to "there's hundreds of IP's rotating and attacking". So the answer to that would be: it depends on who you've pissed off or who's feeling particularly nasty that day. [1]: https://www.cisco.com/c/en/us/products/collateral/security/traffic-anomaly-detector-xt-5600a/prod_white_paper0900aecd8011e927.html https://www.cisco.com/c/en/us/products/collateral/security/t... [2]: http://www.continuitycentral.com/index.php/news/technology/624-the-evolving-nature-of-ddos-attacks-separating-the-wheat-from-the-chaff http://www.continuitycentral.com/index.php/news/technology/6... [3]: https://www.radware.com/newsevents/mediacoverage/changing-challenging-ddos-attacks/ https://www.radware.com/newsevents/mediacoverage/changing-ch...
- viraptor 10y agoIt's non-commital, because at some point, when you have enough zombie hosts properly distributed all over the world attacking you, your only defence is - have more bandwidth than the attackers. If your peers can't filter out the traffic before it hits your network and it simply saturates your pipes, there's nothing you can do inside the company anymore.
- jonahx 10y agoThanks for the reply. Can you elaborate on what you mean by "peers" in the above? Eg, who (or what) would Heroku's peers be?
- throwsep3 10y agoI wonder if this is a diversion to keep Linode's security team busy so they won't notice someone compromising the Xen nodes with XSA-185/6/7/8?
- Scarbutt 10y agoThey switched to KVM.
- HappyTypist 10y agoThis happened last time.
- mappu 10y agoFascinating, seems believable. XSAs are released alarmingly frequently. I've always wondered if KVM is really more secure or if it just gets less scrutiny (certainly there's a class of qemu vulnerabilities that impact both hypervisors).
- rodgerd 10y agoKVM is a much smaller code base, which probably helps.
- technion 10y agoI've noted the AWS security bulletins[0] list nearly every Xen advisory with "AWS customers' data and instances are not affected by these issues, and there is no customer action required." It would appear? you'd need to go back for quite a few months of being unpatched to find a genuine issue. Unless something about Amazon's mitigations don't apply universally. [0] https://aws.amazon.com/security/security-bulletins/ https://aws.amazon.com/security/security-bulletins/
- pmalynin 10y agoThis explains why Package Control, Ansible Docs, etc. were down. Sucks. I knew this has happened before, just didn't know it was Linode.
- chrischen 10y agoDoes the US have a competent cyber-crime division that can handle stuff like this?
- icehawk219 10y agoQuite the opposite. Our government at least appears to work very hard to make sure these attack venues remain wide open.
- SEJeff 10y agoYes, the FBI. They're fantastic at it and part of their job is helping businesses recover from compromise and going after the attackers. However, they're overworked government employees with not enough resources.
- deleted 10y ago[deleted]
- matt_wulfeck 10y agoI think they very much want to, but what can you do if all of your IPs lead to tor exit nodes? From what I've heard, the FBI will collect a whole bunch of information and then sit on their hands because of the above reason.
- 77pt77 10y agoHow can tor nodes be used for that? Tor exposes a SOCKS interface, you can't control TCP/IP with the level of detail needed to perform these attacks.
- gizmo686 10y agoRun the attack itself from a botnet of hacked computers, with the command and control server proxied behind TOR.
- pbhjpbhj 10y ago
- matt_wulfeck 10y agoI disagree with people saying these types of attacks can't be prevented if you switched hosts. I'm sure Google+cloudflare[0] would keep your website online. AWS also if you had the cash. The amount of distributed traffic happening right now against linode would probably only represent a 5% increase in traffic to a popular Google product. At least you know they have the expertise. Nothing against the very smart and talented linode engineers, but the two companies are on very different levels of traffic engineering. [0] https://www.cloudflare.com/google/ https://www.cloudflare.com/google/
- the_duke 10y agoGoogle and AWS probably have sophisticated DDOS mitigation (can anyone comment on this?) and you can scale up pretty quickly. But if your service is the direct target of the attack (as opposed to the whole provider) and your servers are getting hammered... Even if you architecture allows quick horizontal scaling, you still face a tough decision. The attack could go on for days, and the hosting costs can go really high really fast. Which can be catastrophic for a small company.
- user5994461 10y agoAttacks are rarely targeted to the hosting providers. They usually target a specific customer. Google/AWS probably have 100 times the capacity (and redundancy and architecture reliability and failover and awesomeness) of linode. That means that, first, they can't be put down easily, second, a DDoS is limited to a small subset of the infrastructure and doesn't bleed to every customers and services. As for traditional hosting companies (OVH and the likes) When you're being DDoSed, they'll null-route your IP space. (i.e. they advertise your IPs as dont-exist-on-the-internet-anymore). The traffic is dropped while in transit on the internet because it can't go anywhere. It doesn't reach the hosting company anymore. Note: being null-routed means your site and all your services are off the internet and thus effectively dead. As for CloudFlare. They have many locations all around the world and they can absorb a lot of traffic, to the point they themselves cannot be DDoS. They have active monitoring and mitigation against common attacks and known malicious sources, which may prevent the attack without even you knowing about it. When you're under attack, you can block subnet/AS/countries in cloudflare settings, or request a challenge/capcha from every visitors. Cloudflare will reject visitors (with or without challenging them) at their edge location before any traffic can get to you. It is very effective from my experience. Generally speaking. The only way to stop a DDoS is to do it before it reaches your datacenters so you need help from your ISP/provider/CDN. Edit: The attack that put down linode last christmas was against linode itself and not a specific customer. Part of the mitigation included linode moving its critical services behind cloudflare :D
- jtl999 10y agoDigitalOcean has had a few DDoS attacks targeting their SFO1 datacenter over the past few months, but fortunately each one seemed to disappear in under half an hour.
- hehheh 10y agoDigitalOcean is the source of many DDoS attacks. They've got a lot of compromised servers on their network and ignore abuse complaints.
- simonmales 10y agoPlay by Play of Linodes 'twelve day attack' over Christmas and New Year 2015 https://blog.linode.com/2016/01/29/christmas-ddos-retrospective/ https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...
- m0atz 10y agoAny details on the actual size of the attack and the attack vectors used?
- constit-protec 10y agoGodaddy has been getting attacked a lot recently as well. Who is likely behind attacking servers, whether other server companies or governments?
- bogomipz 10y agoOrganized crime syndicates? Vigilante hacktivists?
- hehheh 10y agoGoDaddy, among others, host servers that are a major source of DDoS attacks. I am not sad that they are also victims.
- mikebutash 10y agoWhen you are the dns root record and in many cases hosting too for some 60-70 million domains, someone in the internet is wanting to attack someone at godaddy all the time. There isn't a time they aren't being attacked somehow in all reality, and I'd presume it's much the same for Linode. The means of really combating a ddos is costly and extensive, this is why most use a service like prolexic or silverline, and typically with some massive infrastructures that comes with it. Anything less than n-by 40gb disposable internet pipes, preferably regional as you are, you can/will be smited at will.
- ablagoev 10y agoI've always wondered, while in similar cases GCE/AWS can handle the traffic, is it not chargeable? So, while you will probably not get affected by the DDoS, aren't the costs going to cut your head off?
- technion 10y agoRelated, if the costs reach a point that you cannot pay them, is it not a successful, unmitigated DDoS?
- kalleboo 10y agoIsn't incoming data on AWS free? Or are you thinking of some kind of attack where they're using your infrastructure to amplify outgoing data?
- kalleboo 10y agoIsn't incoming data on AWS free? Or are you thinking of some kind of attack where they're using your infrastructure to amplify outgoing data? Edit: I guess an attack that causes your infrastructure to auto-scale could get expensive REAL quick...
- technion 10y agoI'm running the maths right now.. and you could convince me to take down my side project by just having a server outside their network put wget in a loop targeted at my S3 resources.
- Dowwie 10y agoI have no evidence to support this theory, but I believe that Linode is not an outlier with regards to frequent DDOS attacks. What makes this company special seems to be with how it communicates to its customers when it's under attack. This leads me to wonder: How much do other providers leave customers in the dark?
- bogomipz 10y agoI really feel bad for these folks. Does know if they have a DDOS mitigation strategy other than RTBH with their transit providers? I would have thought that after the 2015 attack they would have looked into traffic scrubbing with something like Arbor Network or Prolexic. I understand that these are not cheap and Linodes as well as many other hosting provider's margins are probably thin but I would think that it would pay for itself in one or two attacks by minimizing customer churn an event like that causes.
- xmatos 10y agoI don't get the hate towards linode here, on hacker news. I've been their client for a couple of years now and I find it an excellent vps provider. Excellent uptime and performance at a pretty good price. AWS has a few outages every year. Google just had one last week. Azure sucks balls. So, why the hate? Is it because it competes with some ycombinator startups?
- fizzbatter 10y agoI don't follow this too closely, so this is just wild speculation from me: But could it simply be severity of the attacks? I keep seeing comments about a 2 week ddos attack last christmas - that's something that i would be shocked to see Google/AWS succumb to. Not that Google/AWS attacks don't happen, i just can't imagine them being down for ~2weeks (I imagine it was just one datacenter from Linode, not the entire service, fwiw)
- warbiscuit 10y agoThey weren't down for the entire two weeks, but various datacenters went up and down for hours, then quieted down for a few days, then was back again, then another hit; stretching across two weeks. One thing that took them so long was that their upstream ISPs at some of the datacenters were themselves unable to handle the DDOS, so they had to switch ISPs, which took a while. I don't see Google/AWS as easy to attack; but I'm not sure why similar tier players like DigitalOcean aren't being hit -- or maybe they're just less transparent about things, or are actually a smaller target (didn't think they were?). edit: here's a postmortem from linode of the christmas attack - https://blog.linode.com/2016/01/29/christmas-ddos-retrospective/ https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...
- hobs 10y agoNo, its because at one time the community actually liked and recommended it and then got burned and acts accordingly. Linode has a bunch of great features, but after seeing them get hacked a half dozen times over really silly things, more DDOS's than you would be happy with, and frankly I have had interactions with their management (just online) and was sorry to have had said interactions. You can also read a bunch of implications from former employees about their management, but you can feel free to discount that given how many times ex-employees are a bit pissed. - Former Linode Customer
- ripken 10y agoYou get what you pay for. TCO of running a highly available data center is more than just the cost of metal, real estate, power, cooling, and cable Going cheap with one of the non-big 3 public cloud providers will cost you.
- ekiara 10y agoI remember a few years ago when I moved my Linode from Fremont to Atlanta to avoid the frequent outages. I've never had show-stopping issues with Linode and the customer service has always been fast and responsive. Now though, I'm thinking of moving to their Frankfurt datacenter. But now I think I need to setup fail-over with another VPS provider. What's a recommended alternative? Is Digital Ocean the next best choice after Linode?
- isarat 10y agoLinode is a cost effective solution comparing to AWS. But the security and DDoS issues could raise eyebrows and bring confidence issues with customers. This has happened even after they posted about enhanced DDoS mitigation strategies like procuring more bandwidth etc. If you look at the history of Linode discussions under HN, most of them were related to DDoS attacks and service downtimes. https://news.ycombinator.com/from?site=linode.com https://news.ycombinator.com/from?site=linode.com I hope they will recover soon and make the service stable.