6 ms·

Glad to hear something official on this...5 or 6 days is way too long to go without something more than "We're working on it" and some light details. I understa

Glad to hear something official on this...5 or 6 days is way too long to go without something more than "We're working on it" and some light details. I understand that it's likely an all-hands-on-deck hair-on-fire situation over there, but those of us who rely on Linode for our own businesses have been largely left in the dark. When our customers are emailing and tweeting us and they just want to know when we are going to be up, and all we can say is "We have no idea, we don't know why this is happening or what's really going on", that's pretty much the definition of a worst case scenario from a customer service standpoint. As someone whose business relies on Linode currently to function, I am sympathetic to Linode's plight...this is the equivalent of someone coming and setting off a bomb in your factory; not exactly something that you can always plan for even if you have prevention measures in place. But they would have kept a lot more of my sympathy long-term if they would have communicated better with their customers in the first place... EDIT: And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...

kunjanshah 11y ago
"It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business." The timing of the DDoS was pretty interesting too, happening when not everyone is available.
- dwightgunning 11y ago
  This was my first thought when I woke up to a bunch of my server monitor alarms at about 4am on Christmas day....
alexforster 11y ago
We know that we've dropped the ball here. To be frank, it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent than "they're attacking our webservers", "they're attacking our core routers", etc. > And it looks like the attackers decided to start things back up again, as Linode.com is unavailable... They're watching our status page for updates and starting new attacks when we resolve previous ones. There's been an almost 1:1 correlation lately.
- silverlight 11y ago
  Indeed, well, keep fighting the good fight. We are cheering for you.
- reefoctopus 11y ago
  Thank you for the update. We really appreciate the information and wish you the best of luck in dealing with this.
- 00deadbeef 11y ago
  I just registered to thank you for the update and to let you know I won't be moving away from Linode as this just gives the attackers what they want.
- SudoAlex 11y ago
  The correlation could also be the fact that you've blocked the attack, they can see that their previous attack is no longer working by testing it themselves - so they'll switch anyway, regardless of a status page update.
- erikpukinskis 11y ago
  > it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent This always rings hollow to me, and yet I hear it over and over. A company like Linode surely has at least a dozen people who can be on call in a situation like this, probably much more. All it takes is for one engineer or even a product person... Heck a technically-minded support person could listen in on the war room meetings and get enough information to post something better than "we're fixing it". 5 minutes of blogging every six hours would be plenty. And yet, people always claim it's impossible and there's no time. Frankly I find it frightening... If you are coding that fast that no one on your team has five minutes to step aside, take a breather after six hours of coding and summarize what the team just spent the last six hours doing, I shudder to think what kind of panicked alarmist interventions you are making. There's no excuse for silence. There just isn't. It's a gross failure on the part of management to prioritize the responsibilities they have to your customers customers. Full stop. Sure, the engineers can't be expected to remember to tap out to blog. But if that's the extent of the accountability structures you can assemble during a crisis, that is a serious organizational failing, particularly for an organization the size of Linode.
nly 11y ago
> As someone whose business relies on Linode currently to function Perhaps its time to consider some failover at another host. Same goes for anyone solely dependent on anyone.
- silverlight 11y ago
  Certainly what we're in the process of doing.
  - mwcampbell 11y ago
    I'm surprised you haven't completely jumped ship already, considering that there are two other providers in particular that offer very similar specs and prices to Linode. Where I work, we moved our primary infrastructure off of Linode the night after the attacks started.
    - workitout 11y ago
      I've tried a lot over the years, no one can compare to the VPS quality Linode provides is what I found.
    - brandon272 11y ago
      All (major) providers suffer from DDoS attacks.
      - tshtf 11y ago
        When was the last successful sustained (1 week plus) CloudFlare, GCE, or AWS DDoS attack?
        brandon272 11y ago
        I have no idea. CloudFlare doesn't offer VPS instances. Google and Amazon are in a league of their own and, last I checked, don't offer the similar services at the same prices with the same level of support and same controls. Amazon might send you a nice bill at the end of the attack depending on what type it was. DigitalOcean, as an example, routinely receives DDoS attacks and will just nullroute your VPS automatically until the attack ends.
Sir_Cmpwn 11y ago
[Disclaimer: Linode employee] Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.
- atmosx 11y ago
  Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?
  - alexforster 11y ago
    Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks. We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.
    - jgord 11y ago
      As a happy linode user, your update is much appreciated. I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires. I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.
      - nickpsecurity 11y ago
        Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm: https://protonmail.com/blog/ddos-protection-guide/ https://protonmail.com/blog/ddos-protection-guide/
brandon272 11y ago
I've seen Linode getting a lot of flack for not updating customers on what is happening, but what clarity does this announcement provide that their DDoS status page wasn't providing? They were updating the status page regularly.
- icelancer 11y ago
  Push, not pull. Send an email. Don't expect people to go to a status page on their own.
  - piquadrat 11y ago
    Which they could easily do by enabling SMS/email/webhook notifications in their statuspage.io settings. As it is, you can only subscribe to an RSS feed it seems.
- Orrfen 11y ago
  It gives those of us hosted on Linode a clear place to point our customers.

kunjanshah 11y ago

"It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business." The timing of the DDoS was pretty interesting too, happening when not everyone is available.

dwightgunning 11y ago
This was my first thought when I woke up to a bunch of my server monitor alarms at about 4am on Christmas day....

alexforster 11y ago

We know that we've dropped the ball here. To be frank, it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent than "they're attacking our webservers", "they're attacking our core routers", etc. > And it looks like the attackers decided to start things back up again, as Linode.com is unavailable... They're watching our status page for updates and starting new attacks when we resolve previous ones. There's been an almost 1:1 correlation lately.

silverlight 11y ago
Indeed, well, keep fighting the good fight. We are cheering for you.
reefoctopus 11y ago
Thank you for the update. We really appreciate the information and wish you the best of luck in dealing with this.
00deadbeef 11y ago
I just registered to thank you for the update and to let you know I won't be moving away from Linode as this just gives the attackers what they want.
SudoAlex 11y ago
The correlation could also be the fact that you've blocked the attack, they can see that their previous attack is no longer working by testing it themselves - so they'll switch anyway, regardless of a status page update.
erikpukinskis 11y ago
> it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent This always rings hollow to me, and yet I hear it over and over. A company like Linode surely has at least a dozen people who can be on call in a situation like this, probably much more. All it takes is for one engineer or even a product person... Heck a technically-minded support person could listen in on the war room meetings and get enough information to post something better than "we're fixing it". 5 minutes of blogging every six hours would be plenty. And yet, people always claim it's impossible and there's no time. Frankly I find it frightening... If you are coding that fast that no one on your team has five minutes to step aside, take a breather after six hours of coding and summarize what the team just spent the last six hours doing, I shudder to think what kind of panicked alarmist interventions you are making. There's no excuse for silence. There just isn't. It's a gross failure on the part of management to prioritize the responsibilities they have to your customers customers. Full stop. Sure, the engineers can't be expected to remember to tap out to blog. But if that's the extent of the accountability structures you can assemble during a crisis, that is a serious organizational failing, particularly for an organization the size of Linode.

nly 11y ago

> As someone whose business relies on Linode currently to function Perhaps its time to consider some failover at another host. Same goes for anyone solely dependent on anyone.

silverlight 11y ago
Certainly what we're in the process of doing.
- mwcampbell 11y ago
  I'm surprised you haven't completely jumped ship already, considering that there are two other providers in particular that offer very similar specs and prices to Linode. Where I work, we moved our primary infrastructure off of Linode the night after the attacks started.
  - workitout 11y ago
    I've tried a lot over the years, no one can compare to the VPS quality Linode provides is what I found.
  - brandon272 11y ago
    All (major) providers suffer from DDoS attacks.
    - tshtf 11y ago
      When was the last successful sustained (1 week plus) CloudFlare, GCE, or AWS DDoS attack?
      - brandon272 11y ago
        I have no idea. CloudFlare doesn't offer VPS instances. Google and Amazon are in a league of their own and, last I checked, don't offer the similar services at the same prices with the same level of support and same controls. Amazon might send you a nice bill at the end of the attack depending on what type it was. DigitalOcean, as an example, routinely receives DDoS attacks and will just nullroute your VPS automatically until the attack ends.

Sir_Cmpwn 11y ago

[Disclaimer: Linode employee] Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.

atmosx 11y ago
Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?
- alexforster 11y ago
  Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks. We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.
  - jgord 11y ago
    As a happy linode user, your update is much appreciated. I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires. I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.
    - nickpsecurity 11y ago
      Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm: https://protonmail.com/blog/ddos-protection-guide/ https://protonmail.com/blog/ddos-protection-guide/

brandon272 11y ago

I've seen Linode getting a lot of flack for not updating customers on what is happening, but what clarity does this announcement provide that their DDoS status page wasn't providing? They were updating the status page regularly.

icelancer 11y ago
Push, not pull. Send an email. Don't expect people to go to a status page on their own.
- piquadrat 11y ago
  Which they could easily do by enabling SMS/email/webhook notifications in their statuspage.io settings. As it is, you can only subscribe to an RSS feed it seems.
Orrfen 11y ago
It gives those of us hosted on Linode a clear place to point our customers.