9 ms·

Linode Manager and API are under attack

68 points by mherrmann 10y ago

hnarayanan 10y ago
This sort of headline involving Linode is (unfortunately) beginning to feel extremely frequent. I am happy that I've since moved away to GCP, but this is still sad to hear.
- charlieegan3 10y ago
  Certainly often seems to come up on HN. https://news.ycombinator.com/from?site=linode.com https://news.ycombinator.com/from?site=linode.com
- cstrat 10y ago
  I feel bad for them and their customers. Wonder what the root cause of all this is, they seem to be a specific target for some reason...
  - boulos 10y ago
    High value (a decent number of high profile customers) and yet easier than say us (Google), AWS or Azure. As the last discussion brought up, I'm curious whether Linode is particularly singled out compared to say Digital Ocean, OVH, Hetzner, et al. but maybe someone like jgrahamc knows?
    - Joe8Bit 10y ago
      Easier in the sense they have less scale/capacity/capability to mitigate the attacks?
      - samhamilton 10y ago
        Yes for sure, think about what Google etc had as teams before they all launched their cloud compute services, I would bet they all had abuse mitigation teams. In this case I would figure there is an overlap of the team and tooling that keeps google.com up and the one that keeps my VM being hosed by a DDoS
- asciihacker 10y ago
  > I am happy that I've since moved away to GCP What is "GCP"?
  - jeffcore 10y ago
    I believe they're referring to Google Cloud Platform.
jupake 10y ago
I'd be interested to know which data centres are being attacked. And why just the api and manager?
- boulos 10y ago
  I don't know where, but as to the why: control planes usually have fairly low traffic (create an instance isn't that frequent per customer) so they'd be easier to swamp if you can get it to consider your traffic. Proper load shedding and rate limiting is kind of hard (you have to decide what you want to happen to well intentioned folks), but anyway that's probably why.
  - andruby 10y ago
    Maybe this attack is part of a plan to get access to the control panel of customers. That would be a way to gain access to some relatively high profile sites.
    - gcb0 10y ago
      more likely just DoSing as a way to prevent people already hacked from bringing hosts down.
- user5994461 10y ago
  > And why just the api and manager? Likely because it's more effective, if you really want to kill the site. Let's say that you find a "/manage/viewall" page which takes some processing to load (more so than the other pages), is never cached, and can't be protected by cloudflare/capcha/other because it must stay open for CLI tools. First, it's easy to overload the site by throwing a couple of expensive requests to a few vulnerable services, rather than to have a botnet flooding 100 Gb/s of traffic to random customer instances. (note: the two strategies are not mutually exclusive). Second, by attacking these services, they impact Linode itself and all customers using it. Someone is really trying to hurt Linode by doing that. The lesson here: There seem to be some naughty attackers who are putting a lot of effort to put Linode down recently... and they seem to succeed to some extent. That really is a bad position for Linode :(
deftnerd 10y ago
My theory is either a ruthless competitor has been trying to destroy linode for years, this is a distraction to cover a different attack on them, or at some point in the past someone attacked and blackmailed them and Linode paid to make it stop. Once you set that president, it's hard to make it stop.
- jacquesm 10y ago
  It's 'precedent' and without any evidence these theories are mostly worthless.
  - deftnerd 10y ago
    You're correct in both regards. As for the first point, that was just an auto correct mistake. My apologies. On the second point, you're correct that my theories are entirely unsubstantiated. I'm just trying to guess why Linode seems to be the victim of these kinds of attacks a lot more than other providers.
    - jacquesm 10y ago
      The first would be fairly unprecedented and would eventually come out, besides, whichever 'ruthless competitor' would initiate it, it would help all the other competitors equally so I think you can simply forget about that one. A 'distraction to cover a different attack': The more obvious explanation is simply to deny control to people who wish to take down or re-image hosts that have already been hacked, that's much more direct, distraction alone of course could be a factor but is less likely if there is a more plausible explanation. Finally, linode is on the record as not paying attackers on principle in spite of being DDOS's within an inch of their lives, so that's a pretty bad thing to say about them unless you have evidence that they did in fact pay attackers in the past. What it could be is a high profile example to scare smaller companies into paying saying 'you don't want to happen to you what happened to Linode'.
      - ryanlol 10y ago
        >Finally, linode is on the record as not paying attackers on principle in spite of being DDOS's within an inch of their lives, so that's a pretty bad thing to say about them unless you have evidence that they did in fact pay attackers in the past. Not DDoS but they've got a history of doing somewhat similar things re: hacks in the past.
alex_hitchins 10y ago
Out of interest, is there anything to suggest this is an outside attack or an inside attack?
- viraptor 10y ago
  outside. 1. "We are currently experiencing a DDoS attack targeting our infrastructure." - you don't call inside attack a DDoS. 2. For internal attacks they can just firewall off / kill offending servers. It would be resolved in minutes rather than "mitigated".
  - alex_hitchins 10y ago
    Thanks, I suppose I should have figure this out. I wondered if there was some attack that could be kicked of within the network that could be more difficult to trace. I guess you could just looks at port traffic at switch level.
xarope 10y ago
I feel bad for Linode. They've always been a good host (I use Linode, as well as Digital Ocean and others), so I would hate to have to switch away.
- prawn 10y ago
  Same. I've found their service to be great and even today they went out of their way to be especially helpful to me with a support query. Can only imagine how much money and time is spent covering for malicious acts like this, malware, viruses, etc.
  - WayneBro 10y ago
    On topic: I feel bad for Linode too. Even though I am not currently using their service, I have used it in the past and I found their tools to be very straightforward and easy to work with. Off topic: > Same. Exactly the same or mostly the same? I am not sure what you mean by that single word sentence. Do you feel the same about Linode? ("bad") Do you use Linode as well as Digital Ocean and others? Would you hate to switch? Or all three...exactly the same? "This." in a response is much maligned around here. I would think the same goes for "Same.". Sorry for getting completely off-topic, but I don't understand why millennials (or other people) are regressing away from fully understandable sentences towards single syllable grunts and crude hieroglyphs. Maybe I'm wrong and you're not a young person - but I assume this is a younger person's habit. Maybe it's not a bad thing and we're all going to evolve psychic powers so that we just know what the other person meant when a single word is spoken. That would be pretty cool.
    - jameskegel 10y ago
      This type of discourse should take place elsewhere.
      - WayneBro 10y ago
        I don't think so. Thanks for your input though!
deleted 10y ago
[deleted]
endijs 10y ago
I'm not really sure why "Linode" and "DDOS" makes the news. It happens regularly. And is nothing to be surprised about. What is new is how they managed to get their systems upgraded to the point where DDOS does not matter anymore. After big blackout (when they were under attack for weeks and even one of their DC's was completely offline for more than a day) their defenses have become way better. Of course - maybe attacks are not so big. I do not know that. It's my 8th year with them. During those years only once (I think) I had downtime because of those DDOS attacks. Not bad at all.
- matt_wulfeck 10y ago
  It's a huge deal for a linode customer. It's not like your site is DDoS'd and you can't access. You can't access anything on linode when their entire service is DDoS'd. Part of the reason I think it's news is to shame linode. The likelihood of a DDoS attack bringing down the AWS management console is extremely low (has it ever happened?) Yet attackers have managed to do it multiple times on Linode.
  - endijs 10y ago
    I'm one of those customers (for 8th year already) and for me it's not a big deal. Unless you are one of those high profile customers who needs constant API access, those few minutes of downtime is not a big deal. Not everyone is spinning up and down new instances all the time. Plus - not all attacks result in downtime for them. Sometimes you do not notice at all, sometimes responsiveness will suffer a bit. And as I said - their defenses have become way better than they were a year ago. And it's not really fair to compare Linode to AWS. It would be more interesting to see how would DigitalOcean handle the same amount of attacks. And I doubt DO would handle them any better than Linode.
  - oliwarner 10y ago
    Speaking as a customer, I've never actually seen any direct service downtime as a result of a DDOS. Specific services (usually their web manager) have gone down but that hasn't affected my clients' sites running on my servers. Obvious scope for improvement —and I've no doubt this is a constant engineering sink— but not being able to commission new nodes through the web interface for a few hours hasn't yet ruined my day.
throw2016 10y ago
It feels off that Linode is being targetted so intensely and one feels bad for them but this can destroy their business. It will be a pity if the barriers to entry are being constantly raised by the need for more and more mitigation and this will reflect in the level of expertise needed and prices for end consumers, and even then with the constant threat of blackouts. It also becomes easy to silence inconvenient voices. It is difficult to run a business or website if you need to constantly fight extortionists or make payoffs to address threats of downtime. Today Linode is in the news, tomorrow it could be anyone else targetted. There is no easy solution here, without putting constraints somewhere but there must be a way to make the web more resilient and robust and not subject to the whims of extortionists and other malicious agents.
alvil 10y ago
Qui bono? :)
rebootthesystem 10y ago
Linode customer here. Going to take a tangent. Way back when, Toyota instituted a policy that required every line worker to effectively stop the factory if a defect was found. Engineers would come to the floor. They were tasked with creating a permanent solution for the problem, one that would have that never happen again. For the first few months Toyota had a very hard time getting cars out the door. As time went by things got better. Eventually they got to the point where nearly every car coming off the line was perfect. Around the same time companies like Mercedes Benz were devoting no less than 25% of their factory floor towards fixing manufacturing defects. During this same time period Toyota was devoting less than 5% of their factory floorplan to fixes. This, ironically, meant that the Japanese company was probably delivering a higher quality and more reliable product than the high end car manufacturer. Source: Read a book on the subject many years ago: "The Machine that changed the world" https://goo.gl/VQw6HS https://goo.gl/VQw6HS Back to Linode. The fact that they've seen so many attacks gives them the opportunity to become far better when compared to someone who never sees attacks. Whether they go there or not is entirely in their hands. If this is what's happening they need to take the time to communicate it to the world. There are at least two possible attitudes: The first is to live in fear of attacks and hope they never happens. The second is to embrace them as an opportunity to get better. The latter is a formula for disaster, or mediocrity at best. The latter is supported by millions of years of history on this planet showing that things get better because something or someone had to find a solution to a problem. You can't become a good sailor without facing a few storms. I'm sticking with Linode.
- sb8244 10y ago
  The former is a recipe for disaster.
  - rebootthesystem 10y ago
    Darn! Too late to edit. Thanks.
ksec 10y ago
It cant be a customers of Linode that is being targetted, Linode has seen WAY MORE DDOS attack then any other web host. Someone must have hated them or a competitor is trying to destroy them. These frequent attack just isn't normal.
alexforster 10y ago
I appreciate the kind words in many of the comments here, and I'm sure my colleagues do too. For reference, our third party monitoring measured approximately 23 minutes of downtime– 1m 46s @ 2016-09-08 01:37:53 EDT 1m 10s @ 2016-09-08 01:38:14 EDT 17m 4s @ 2016-09-08 01:40:28 EDT 2m 11s @ 2016-09-08 01:58:29 EDT We are now capable of absorbing most volumetric attacks toward our web infrastructure. This outage was a layer 7 attack that needed to be manually mitigated.
vorotato 10y ago
I wonder if it has to do with people using linode as a way to get around a particular country's censorship.

68 points by mherrmann 10y ago

This sort of headline involving Linode is (unfortunately) beginning to feel extremely frequent. I am happy that I've since moved away to GCP, but this is still sad to hear.

charlieegan3 10y ago
Certainly often seems to come up on HN. https://news.ycombinator.com/from?site=linode.com https://news.ycombinator.com/from?site=linode.com
cstrat 10y ago
I feel bad for them and their customers. Wonder what the root cause of all this is, they seem to be a specific target for some reason...
- boulos 10y ago
  High value (a decent number of high profile customers) and yet easier than say us (Google), AWS or Azure. As the last discussion brought up, I'm curious whether Linode is particularly singled out compared to say Digital Ocean, OVH, Hetzner, et al. but maybe someone like jgrahamc knows?
  - Joe8Bit 10y ago
    Easier in the sense they have less scale/capacity/capability to mitigate the attacks?
    - samhamilton 10y ago
      Yes for sure, think about what Google etc had as teams before they all launched their cloud compute services, I would bet they all had abuse mitigation teams. In this case I would figure there is an overlap of the team and tooling that keeps google.com up and the one that keeps my VM being hosed by a DDoS
asciihacker 10y ago
> I am happy that I've since moved away to GCP What is "GCP"?
- jeffcore 10y ago
  I believe they're referring to Google Cloud Platform.

jupake 10y ago

I'd be interested to know which data centres are being attacked. And why just the api and manager?

boulos 10y ago
I don't know where, but as to the why: control planes usually have fairly low traffic (create an instance isn't that frequent per customer) so they'd be easier to swamp if you can get it to consider your traffic. Proper load shedding and rate limiting is kind of hard (you have to decide what you want to happen to well intentioned folks), but anyway that's probably why.
- andruby 10y ago
  Maybe this attack is part of a plan to get access to the control panel of customers. That would be a way to gain access to some relatively high profile sites.
  - gcb0 10y ago
    more likely just DoSing as a way to prevent people already hacked from bringing hosts down.
user5994461 10y ago
> And why just the api and manager? Likely because it's more effective, if you really want to kill the site. Let's say that you find a "/manage/viewall" page which takes some processing to load (more so than the other pages), is never cached, and can't be protected by cloudflare/capcha/other because it must stay open for CLI tools. First, it's easy to overload the site by throwing a couple of expensive requests to a few vulnerable services, rather than to have a botnet flooding 100 Gb/s of traffic to random customer instances. (note: the two strategies are not mutually exclusive). Second, by attacking these services, they impact Linode itself and all customers using it. Someone is really trying to hurt Linode by doing that. The lesson here: There seem to be some naughty attackers who are putting a lot of effort to put Linode down recently... and they seem to succeed to some extent. That really is a bad position for Linode :(

deftnerd 10y ago

My theory is either a ruthless competitor has been trying to destroy linode for years, this is a distraction to cover a different attack on them, or at some point in the past someone attacked and blackmailed them and Linode paid to make it stop. Once you set that president, it's hard to make it stop.

jacquesm 10y ago
It's 'precedent' and without any evidence these theories are mostly worthless.
- deftnerd 10y ago
  You're correct in both regards. As for the first point, that was just an auto correct mistake. My apologies. On the second point, you're correct that my theories are entirely unsubstantiated. I'm just trying to guess why Linode seems to be the victim of these kinds of attacks a lot more than other providers.
  - jacquesm 10y ago
    The first would be fairly unprecedented and would eventually come out, besides, whichever 'ruthless competitor' would initiate it, it would help all the other competitors equally so I think you can simply forget about that one. A 'distraction to cover a different attack': The more obvious explanation is simply to deny control to people who wish to take down or re-image hosts that have already been hacked, that's much more direct, distraction alone of course could be a factor but is less likely if there is a more plausible explanation. Finally, linode is on the record as not paying attackers on principle in spite of being DDOS's within an inch of their lives, so that's a pretty bad thing to say about them unless you have evidence that they did in fact pay attackers in the past. What it could be is a high profile example to scare smaller companies into paying saying 'you don't want to happen to you what happened to Linode'.
    - ryanlol 10y ago
      >Finally, linode is on the record as not paying attackers on principle in spite of being DDOS's within an inch of their lives, so that's a pretty bad thing to say about them unless you have evidence that they did in fact pay attackers in the past. Not DDoS but they've got a history of doing somewhat similar things re: hacks in the past.

alex_hitchins 10y ago

Out of interest, is there anything to suggest this is an outside attack or an inside attack?

viraptor 10y ago
outside. 1. "We are currently experiencing a DDoS attack targeting our infrastructure." - you don't call inside attack a DDoS. 2. For internal attacks they can just firewall off / kill offending servers. It would be resolved in minutes rather than "mitigated".
- alex_hitchins 10y ago
  Thanks, I suppose I should have figure this out. I wondered if there was some attack that could be kicked of within the network that could be more difficult to trace. I guess you could just looks at port traffic at switch level.

xarope 10y ago

I feel bad for Linode. They've always been a good host (I use Linode, as well as Digital Ocean and others), so I would hate to have to switch away.

prawn 10y ago
Same. I've found their service to be great and even today they went out of their way to be especially helpful to me with a support query. Can only imagine how much money and time is spent covering for malicious acts like this, malware, viruses, etc.
- WayneBro 10y ago
  On topic: I feel bad for Linode too. Even though I am not currently using their service, I have used it in the past and I found their tools to be very straightforward and easy to work with. Off topic: > Same. Exactly the same or mostly the same? I am not sure what you mean by that single word sentence. Do you feel the same about Linode? ("bad") Do you use Linode as well as Digital Ocean and others? Would you hate to switch? Or all three...exactly the same? "This." in a response is much maligned around here. I would think the same goes for "Same.". Sorry for getting completely off-topic, but I don't understand why millennials (or other people) are regressing away from fully understandable sentences towards single syllable grunts and crude hieroglyphs. Maybe I'm wrong and you're not a young person - but I assume this is a younger person's habit. Maybe it's not a bad thing and we're all going to evolve psychic powers so that we just know what the other person meant when a single word is spoken. That would be pretty cool.
  - jameskegel 10y ago
    This type of discourse should take place elsewhere.
    - WayneBro 10y ago
      I don't think so. Thanks for your input though!

deleted 10y ago

[deleted]

endijs 10y ago

I'm not really sure why "Linode" and "DDOS" makes the news. It happens regularly. And is nothing to be surprised about. What is new is how they managed to get their systems upgraded to the point where DDOS does not matter anymore. After big blackout (when they were under attack for weeks and even one of their DC's was completely offline for more than a day) their defenses have become way better. Of course - maybe attacks are not so big. I do not know that. It's my 8th year with them. During those years only once (I think) I had downtime because of those DDOS attacks. Not bad at all.

matt_wulfeck 10y ago
It's a huge deal for a linode customer. It's not like your site is DDoS'd and you can't access. You can't access anything on linode when their entire service is DDoS'd. Part of the reason I think it's news is to shame linode. The likelihood of a DDoS attack bringing down the AWS management console is extremely low (has it ever happened?) Yet attackers have managed to do it multiple times on Linode.
- endijs 10y ago
  I'm one of those customers (for 8th year already) and for me it's not a big deal. Unless you are one of those high profile customers who needs constant API access, those few minutes of downtime is not a big deal. Not everyone is spinning up and down new instances all the time. Plus - not all attacks result in downtime for them. Sometimes you do not notice at all, sometimes responsiveness will suffer a bit. And as I said - their defenses have become way better than they were a year ago. And it's not really fair to compare Linode to AWS. It would be more interesting to see how would DigitalOcean handle the same amount of attacks. And I doubt DO would handle them any better than Linode.
- oliwarner 10y ago
  Speaking as a customer, I've never actually seen any direct service downtime as a result of a DDOS. Specific services (usually their web manager) have gone down but that hasn't affected my clients' sites running on my servers. Obvious scope for improvement —and I've no doubt this is a constant engineering sink— but not being able to commission new nodes through the web interface for a few hours hasn't yet ruined my day.

throw2016 10y ago

It feels off that Linode is being targetted so intensely and one feels bad for them but this can destroy their business. It will be a pity if the barriers to entry are being constantly raised by the need for more and more mitigation and this will reflect in the level of expertise needed and prices for end consumers, and even then with the constant threat of blackouts. It also becomes easy to silence inconvenient voices. It is difficult to run a business or website if you need to constantly fight extortionists or make payoffs to address threats of downtime. Today Linode is in the news, tomorrow it could be anyone else targetted. There is no easy solution here, without putting constraints somewhere but there must be a way to make the web more resilient and robust and not subject to the whims of extortionists and other malicious agents.

alvil 10y ago

Qui bono? :)

rebootthesystem 10y ago

Linode customer here. Going to take a tangent. Way back when, Toyota instituted a policy that required every line worker to effectively stop the factory if a defect was found. Engineers would come to the floor. They were tasked with creating a permanent solution for the problem, one that would have that never happen again. For the first few months Toyota had a very hard time getting cars out the door. As time went by things got better. Eventually they got to the point where nearly every car coming off the line was perfect. Around the same time companies like Mercedes Benz were devoting no less than 25% of their factory floor towards fixing manufacturing defects. During this same time period Toyota was devoting less than 5% of their factory floorplan to fixes. This, ironically, meant that the Japanese company was probably delivering a higher quality and more reliable product than the high end car manufacturer. Source: Read a book on the subject many years ago: "The Machine that changed the world" https://goo.gl/VQw6HS https://goo.gl/VQw6HS Back to Linode. The fact that they've seen so many attacks gives them the opportunity to become far better when compared to someone who never sees attacks. Whether they go there or not is entirely in their hands. If this is what's happening they need to take the time to communicate it to the world. There are at least two possible attitudes: The first is to live in fear of attacks and hope they never happens. The second is to embrace them as an opportunity to get better. The latter is a formula for disaster, or mediocrity at best. The latter is supported by millions of years of history on this planet showing that things get better because something or someone had to find a solution to a problem. You can't become a good sailor without facing a few storms. I'm sticking with Linode.

sb8244 10y ago
The former is a recipe for disaster.
- rebootthesystem 10y ago
  Darn! Too late to edit. Thanks.

ksec 10y ago

It cant be a customers of Linode that is being targetted, Linode has seen WAY MORE DDOS attack then any other web host. Someone must have hated them or a competitor is trying to destroy them. These frequent attack just isn't normal.

alexforster 10y ago

I appreciate the kind words in many of the comments here, and I'm sure my colleagues do too. For reference, our third party monitoring measured approximately 23 minutes of downtime– 1m 46s @ 2016-09-08 01:37:53 EDT 1m 10s @ 2016-09-08 01:38:14 EDT 17m 4s @ 2016-09-08 01:40:28 EDT 2m 11s @ 2016-09-08 01:58:29 EDT We are now capable of absorbing most volumetric attacks toward our web infrastructure. This outage was a layer 7 attack that needed to be manually mitigated.

vorotato 10y ago

I wonder if it has to do with people using linode as a way to get around a particular country's censorship.