16 ms·
Linode is suffering on-going DDoS attacks
- silverlight 11y agoBeen happening daily for about 5 days now. Pretty frustrating but I'm sure it's frustrating for them too. Thinking about moving to Google Compute Engine instead.
- deleted 11y ago[deleted]
- empressplay 11y agoI worry that moving to a larger company will just hasten the consolidation of hosting to a few players.
- deleted 11y ago[deleted]
- nothrabannosir 11y agoHe(/she?) never said he wouldn't do it, just that it worried him. Back seat doesn't mean thrown out of the car. People need to keep bringing this up, lest we completely forget about it and wake up when it's too late.
- swiley 11y agoThe best thing for reliability is to use multiple companies. AWS has proved that multiple times.
- yeukhon 11y agoDisagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to use AWS, please build on multiple regions.
- mwcampbell 11y agoI have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.
- yeukhon 11y agoHuman capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also. There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for tax purpose). Small to medium usually don't have time to deal with multiple vendors. Try to be vendor agnostic is really great but again, even in the world of open stack, you either manage all of your open stack installation yourself (which is expensive), or you end up one vendor running one version behind, or not offering 100% feature. I think it is more worthwhile to first complete automation and self-healing in one vendor, before thinking leaping to another one. It took so many engineers at Netflix to build a resilient AWS environments. VMs go down all the time, more often than AWS pushing some bad commits and cause half of their native services go down. There is so much more to engineer in other areas and those are more likely to put you out of service first, so unless you want to all your team dealing with fires every day.... we were putting band-aids together most of the time.
- Dylan16807 11y ago12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.
- 11y ago
- albertoleal 11y agoHow does one set up to use multiple vendors?
- ryanlol 11y agoYou buy servers from multiple different companies, you install software on them.
- albertoleal 11y agoAnyone can do that. I was actually wondering if there are software to automate this.
- ryanlol 11y agoOpenstack might allow you to do this. (I haven't used it myself.)
- ultramancool 11y agoAnsible is probably the best bet - once you have SSH it'll pretty much work with anything and it also has support for creation of servers on several cloud providers and stuff. We use it for automatic configuration of everything from colocated hardware to $5/mo VMs on DigitalOcean and other low-cost virtual server providers and it works great.
- rubiquity 11y agoYou could use something like Terraform to help automate this. It's like CloudFormation but also supports GCE, Azure, DigitalOcean, etc. 0 - https://terraform.io/ https://terraform.io/
- mcone 11y agoYou might consider taking a look at http://www.cloud66.com/ http://www.cloud66.com/
- doublerebel 11y ago
- beachstartup 11y agothis is going to happen no matter what - the rate of consolidation in my opinion is actually accelerating. despite what anyone says, the vast majority of people choose their hosting provider in order to "not get fired", i.e. they are extremely risk-averse, to the point of irrationality, even when presented with massive savings, better service, better underlying hardware and network, and more transparency. most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers). at the end of the day it's a server sitting in a rack in a datacenter, connected to ethernet. beyond a certain level of quality (tier 3 dc, server class hardware, an enterprise quality network) it's really all the same. people should choose their hosting provider on 1. quality of implementation 2. price and 3. whether or not the provider actually gives a shit about you and your account, but they usually just go with the name, like many other markets.
- yeukhon 11y ago> most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers). Even AWS can charge you big time. As I am getting more and more familiar with AWS every day, the #1 thing on my list going forward is to to sit down with your TAM and organize architecture review. There are services on AWS lacking completeness and can bite you in the end if you go straight with it without knowing what you are getting into. Disclaimer: I like AWS pretty well.
- jedicoffee 11y ago:(
- mwcampbell 11y agoYep, this started on Friday (Christmas day). I assume Linode itself is the target of the attacks, since they have spanned multiple regions.
- empressplay 11y agoAn extortion attempt, I suspect.
- timdorr 11y agoDallas is their oldest DC, so they have the most customers there and therefore the most attack surface area at that location. It could just as easily be one of their customers.
- dangrossman 11y agoThe DDOS has hit their data centers in Dallas, Fremont, London, Newark, Atlanta, and Singapore since Christmas.
- guelo 11y agoI wonder if the attackers chose Christmas day in order to ensure that there were fewer Linode engineers available to stop the attack.
- scurvy 11y agoCan confirm that the attacks started Friday. We got alarms for some tertiary services we run in their datacenters (multiple geos).
- streetracer23 11y agoPerforming such large attacks for 5 days in a row costs thousands dollars to the attackers. It is not something a kid can do from their bedroom. Only a big company can have the budget to pay thousands to perform a 5-day multi Gbps attack against another company. Here it is clear that there is a competitor attacking Linode. A competitor who wants to extort them.
- Hello71 11y agoLink says DoS, not DDOS (which isn't even a proper acronym).
- robandrews 11y agoDoS == Denial of Service; DDoS == Distributed Denial of Service
- jafingi 11y agoMy four London servers were also unreachable last week. Looks like Linode have a pretty rough time.
- mmaunder 11y agoSo weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.
- alexandrerond 11y agoI am not sure there is a magic way to mitigate these attacks permanently other than playing cat and mouse until the attacker gets tired. Linode has in fact mitigated the attacks, only to find them pointing to a different part of their infra or DCs. At this point, it is probably a good sign that they haven't gone completely down (for long). Current tooling and botnets make DDoS-ing fairly easy today, despite countermeasures.
- kyrra 11y agoIf the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.
- mmaunder 11y agoThat's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing? Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.
- at-fates-hands 11y agoLooks like they have a history of suffering these kinds of attacks: (2012) Upcoming DDOS Attack - FINAL Warning - https://forum.linode.com/viewtopic.php?t=8530 https://forum.linode.com/viewtopic.php?t=8530 (2013) Linode Mitigates DDoS Attack on Linode Manager - http://www.thewhir.com/web-hosting-news/linode-mitigates-ddos-attack-on-linode-manager http://www.thewhir.com/web-hosting-news/linode-mitigates-ddo... (July 2015) Incident Report for Linode - http://status.linode.com/incidents/vnslh3rmm9gq http://status.linode.com/incidents/vnslh3rmm9gq So what makes them such an attractive target for these types of attacks?
- xena 11y agoThe fact that so many people on their IRC channel will react to the attacks.
- scurvy 11y agoLOL at the "proof" attack. "We have a huge 500gbps cannon! But to show we're serious, here's a 2gbps attack. Take that!" That's hilarious. I wouldn't have taken it seriously. I also doubt that the group that threatened them is the same group that is attacking them. I bet it's at least 1 other group. 2gbps. I doubt that even registered on their bandwidth graphs.
- workitout 11y agoAnd Google and AWS and Cloudflare and DigitalOcean and every other VPS provider.
- hacker341 11y agoI am a hacker----i am a hacker--- i an a hackeri imi voi redeschide blogul(hackerdecodequatre.ulcraft.com) in 2017 --vo ataca conturi sitr-uri sociale --- voi sparge site utile institutiilor inportante si le voi publica pe blogul meu
- tshtf 11y agoJust 2 days ago, a Linode employee was badmouthing AWS here on HN for being too expensive: https://news.ycombinator.com/item?id=10796094 https://news.ycombinator.com/item?id=10796094 A DDoS will be much more expensive to customers than choosing AWS over Linode (or an equivalent low-priced service). EC2 has been around since 2006, and never has had any issues resembling this.
- scrollaway 11y agoSo you're comparing DDoS of a provider with everyday spending? I was contracted a few months ago to save a website crumbling under its bills. They had $11k / month in AWS bills. Brought it down to $600 / month by switching them to Linode with a more reasonable stack. Not everything is a nail. Sometimes the mistake is choosing AWS.
- cynicalkane 11y agoIf you reduced their bill from $11k to $600, I'm betting AWS was not the main problem.
- workitout 11y agoGiven that the price calculators for Google Cloud, HP Cloud, Rackspace, etc are online, you can check for yourself. The key one is bandwidth cost for their services.
- eric_h 11y agoAmusingly, we moved our infrastructure from Linode to AWS and cut our bill roughly in half about a year ago.
- scrollaway 11y agoThat can happen as well of course. Like I said, not everything is a nail. Sometimes AWS is the wrong choice, sometimes it's the right choice. However a lot of people jump on AWS when they really should not.
- thrownaway2424 11y agoI would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.
- mwcampbell 11y agoWhat about DigitalOcean? Its pricing is comparable to Linode's, yet DigitalOcean is now the second largest web host in the world according to Netcraft. Do you still think that AWS, Google Cloud Platform, and Microsoft Azure offer better protection?
- deno 11y agoOVH offers very comprehensive DDoS protection with all but the most budget servers: https://www.ovh.com/us/anti-ddos/ https://www.ovh.com/us/anti-ddos/
- MichaelGG 11y agoComments from people seem to indicate their anti DDoS was "we blackhole you if you draw an attack". Has that changed?
- deno 11y agoYes. This hasn’t been true for a long time. OVH runs a huge network and they invested in tools to mitigate and neutralize DDoS attacks[1]. Of course an attack can still saturate your servers’ NICs but they won’t drop you anymore. At their scale they are basically forced to handle big attacks on regular basis. The fact that they offer this protection in their basic package is what makes them a great host[2]. They don’t get a fraction of the love Cloudflare gets, but they should. [1] https://www.ovh.com/us/anti-ddos/hoovering-up.xml https://www.ovh.com/us/anti-ddos/hoovering-up.xml [2] https://www.ovh.com/us/news/articles/a1171.protection-anti-ddos-service-standard https://www.ovh.com/us/news/articles/a1171.protection-anti-d...
- mikegioia 11y agoThis may be a dumb question as I haven't read the bill, but isn't this what CISA is supposed to help guard against in the future? If Linode has an easier way of sharing info with the USG, can that help pinpoint and mitigate this attack from happening in the future? Edit: how about a response instead of downvotes?
- pjc50 11y agoDealing with DDOS has always been way below anti-piracy on in the priority of the relevant US authorities. It's very easy to do and hard to identify those ultimately responsible. Although every now and again a big botnet command-and-control network gets shut down. Nobody's interested in the defensive side of ""cyberwarfare"".
- fanf2 11y agoBotnet takedowns seem to be mostly due to industry cooperation without significant help from the cops.
- MichaelGG 11y agoDDoSes are easy because ISPs don't want to filter traffic (BCP 38). If ISPs did this, then it wouldn't be so easy to spoof source IPs and much easier to block or track down offenders. I don't see how CISA is gonna change this. If the USG wanted to stop this stuff, they'd create a policy forcing ISPs to filter. Same as VoIP and fake caller ID. If the FCC really cared, they'd start fining.
- snowwrestler 11y agoCISA is really intended to address sophisticated attacks that depend on vulnerabilities or vectors, and therefore have distinct signatures. These signatures are what companies will share with the government and each other. A DDOS is usually just a dumb flood of traffic. There's no signature to share, and blocking IP addresses is usually pointless since big DDOS's come from large networks of hijacked personal computers.
- altern8 11y agoFor a change... :-/
- peterhadlaw 11y agoIt's funny that it took an event like this for me to realize this but now I see that it's really a bunch of companies (like Linode) all leasing out space in some huge data-center. I use ASO vps hosting and they too have been experiencing a large network outage. I'm sure you can find at least a handful of hosting companies that are SOL right now too. I guess I understand the whole premise of putting all your eggs into one basket, but then again you really shouldn't.
- gtrubetskoy 11y agoI used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't as distributed as it may seem - it can just be a couple of machines on a very well connected network (e.g. a university). But getting a hold of someone in the middle to filter that traffic can take a very long time or be outright impossible. First-responder network engineers (typically referred to as "security") are overworked and underqualified, and the people who really know their stuff typically can't be bothered with silly DOS attacks. We've also observed that (D)DOS's happen because of content. Anything political, religious, or whatever other shade of the many things someone out there disapproves of is a potential target for a DOS. Contrary to what you may read in the press, extortion is only a small minority of all DOS attacks out there. We've actually told customers to go away because their content was too DOS-prone. And because these things usually happen across countries, even though they are very real crimes that cause serious damage and cost money, they are hardly ever prosecuted. As the target of a DOS all you want is for it to stop, nobody ever bothers reporting it to the authorities afterwards (because how would you even know who the "authority" is).
- devicenull 11y ago> The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. These days it's the hundreds of thousands of misconfigured NTP servers, recursive DNS servers, and various other protocols being abused for reflection attacks. Granted, it still requires that the attacker have the ability to spoof packets, but preventing that requires even more time investment and has very little benefit to the ISP.
- jsmthrowaway 11y agoBCP 38 is like herd immunity with immunization, and much like anti-vaccine folks, networks that don't follow it are knowingly choosing to infect people for any number of reasons. Despite your claim, it is extremely easy to implement and has been a known best practice, with accompanying educational Web sites devoted to the topic[0], for many years. There are nearly zero reasons for your AS to transmit forged packets, and if your configuration allows it, you are knowingly making the Internet a worse place with your laziness and transferring your laziness to other people like me that carry pagers. This isn't a surprise to any network administrator unless they've spent their entire career not reading RFCs. I'm of the opinion that networks that allow customers to emit forged source addresses should be depeered until they take the literal hour to fix it. "But we have to update equipment when we get new blocks!" Boo hoo. Automate it or get off the Internet so I can stop spending my life dealing with your customer's amplified traffic. If you run a non-transit/eyeball AS, you are in the absolute best position to stop these types of attacks from ever happening. The rest of the Internet, particularly your transit peers, can't really clean up after you on this one. Do us all a favor. [0]: http://bcp38.info http://bcp38.info
- mizchief2 11y agoYet the US socialist say a flat tax with very few exceptions unfairly favors the rich....
- Keverw 11y agoOh wow. I remember a couple months ago the ATL datacenter had network issues too. Really annoying, but I guess it's not their fault 100%. I wish they offered more DDoS protection solutions. I know some VPS companies specialize in that offering for things like game servers. It'd be nice if some sort of solution could just be included. I don't know if it's more of a technical issue or legal problem. As far as I know the only way is to null route(which is an inconvenience to the website, but no traffic going to the router other client's are on) or just adding a bunch of servers to "soak" up the extra bandwidth.
- X-Istence 11y agoOVH does VPS's and they have their own Anti-DDoS network setup that is pretty amazing: https://www.ovh.com/us/anti-ddos/ https://www.ovh.com/us/anti-ddos/
- Keverw 11y agoInteresting. What is "Multi-point Mitigation"? I know it mentions a few locations. I googled it and it just brings me back to that page. I wonder if any solution would shutdown a VM and then restart it on another host but that'd be really sucky in some situations like an app might not shutdown cleanly, or the app is in the middle of something like charging a credit card.
- X-Istence 11y agoMulti-point just means that the traffic is going to flow to the nearest datacenter that hosts the VAC, gets filtered and checked and then traverses on to your system. The goal is to spread the DDoS out over as many different bandwidth heavy locations as possible. See the images at the bottom of this page: https://www.ovh.com/us/anti-ddos/hoovering-up.xml https://www.ovh.com/us/anti-ddos/hoovering-up.xml
- workitout 11y agoI have most OVH IP's blocked on my servers because I get too many hack attacks from VMs they are hosting for their customers.
- TheWoodsy 11y agoI wonder if this is correlated to the extortion thread on NANOG I saw the other day.
- latenightcoding 11y agoStrangely this whole thread convinced me to try Linode. Why? everyone keeps saying it's so cheap and I do A LOT of web crawling so I don't need 99.999% uptime
- blisterpeanuts 11y agoI use it for basic stuff, mainly for testing node apps, no high throughput. It's got a nice, simple management console. I don't like the fact that apt-get operations to set up your VM can blow your monthly quota pretty fast if you're not careful, but other than that, it's been fine. Periodically, they double everyone's bandwidth and storage for free; it's something to look forward to in another year or two... I hope.... if these DDOS idiots don't ruin it for everyone.
- deleted 11y ago[deleted]
- stefantalpalaru 11y ago> I don't like the fact that apt-get operations to set up your VM can blow your monthly quota pretty fast if you're not careful What are you talking about? From https://www.linode.com/pricing https://www.linode.com/pricing : "Please note that all inbound traffic is free and will not count against your quota."
- jonathanoliver 11y agoAny the distros are configured to use local Linode mirrors for apt-get anyway.
- stefantalpalaru 11y agoAll the inbound traffic, not just traffic inside the Linode network. "Inbound" means from the Internet to the Linode-hosted server.
- USNetizen 11y agoI find this ironic because about 2 years ago I had a couple VMs with them that suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a whole other platform. Linode, on the other hand, simply blamed us for supposedly causing the repeat DDoS attacks - one after another for months on end. They even got rather flippant with me exclaiming how "dumb" I was to not understand that is was MY problem apparently, not theirs, even though the target of the DDoS was several addresses in their IP block and not tied to a single client domain, server or anything else. They continued to blame us personally, and even tried to get us to foot the $3,000+ traffic bill. Well, the tables have turned. Looks like it IS a Linode problem after all. And not a one-time issue, either. This has been happening to them for YEARS.
- huac 11y ago+1 for IP block DDOS'ing - last time I ran a Linode VM it was DDOS'd to smitheereens within half an hour of putting it up (and of course, without any notice that I was liable for overages...)
- ryukafalz 11y ago>(and of course, without any notice that I was liable for overages...) Inbound traffic does not count against your bandwidth quota, so if you were the target of a DoS attack, this would not result in overages. https://www.linode.com/pricing https://www.linode.com/pricing (Unless you were responding to all that traffic, that is.)
- USNetizen 11y agoThey were DDoSing HTTP amongst other services, so, yeah, there was a response via the websites being hosted. That's where the overages came from.
- workitout 11y agoEvery point you make I can counter, I've been with them for years, have multiple VMs with them, find their support team fast, competent and courteous. DDOS is a problem for every ISP and every ISP customer.
- rdl 11y agoIf anyone from Linode admin team would like some help analyzing the attack/friendly advice on mitigations, please contact me (or anyone else at CloudFlare); we see a lot of these.
- imoff 11y agoHow do you guys trace the real packet sender of a packet with a spoofed IP address?
- toomuchtodo 11y agoClients that don't interpret javascript, typically.
- rdl 11y agoWe don't, generally. It would be challenging. If it were a volumetric attack, you could walk back links to find the source. But for anything but a huge attack which lasted for weeks/months, that would be inefficient use of resources. Paul Vixie is really at the forefront of pushing for providers to solve this problem. Until that happens (and they've tried for a decade), it will remain technically difficult/impossible, so you need to use other solutions to mitigate attacks.
- technikempire 11y agoWhenever I see top level comments where people are saying "this is what you get for not going with X", I am more and more convinced that a competitor is doing this. I for one am not leaving Linode. I JUST recommended to several new clients to setup at Linode and I've already had to explain that server setup is halted because of this attack. Did I recommend moving away from Linode? Nope. Did I hang my head in shame or stutter when I delivered the news, feeling it would reflect poorly on me or Linode? Nope. I told my clients this is the best place for them and I stick by it. Just because there are scumbags specifically attacking your service right now, doesn't mean another service is better, or that your service is poor. It just means that hey, news flash, there are scumbags in the world.
- jafingi 11y agoLinode is _still_ hit by DoS. They currently have a "Major Outage" in the London DC. Seems like there are issues in almost all DC's. Unbelievable they can't get that DoS stopped! My server have had 19 outages the past 7 days, and over 5 hours of downtime! Still no e-mail from Linode whatsoever..
- pbowyer 11y agoI know. I didn't mind when the first outage happened in London (24th?) - these things happen. But now... this is getting beyond a joke. I'm going to have to consider moving, which as I host for friends is not as straightforward as it should be (since they control the DNS)