5 ms·

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documenta

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

alexandrerond 11y ago
I am not sure there is a magic way to mitigate these attacks permanently other than playing cat and mouse until the attacker gets tired. Linode has in fact mitigated the attacks, only to find them pointing to a different part of their infra or DCs. At this point, it is probably a good sign that they haven't gone completely down (for long). Current tooling and botnets make DDoS-ing fairly easy today, despite countermeasures.
kyrra 11y ago
If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.
- mmaunder 11y ago
  That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing? Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.
  - lftl 11y ago
    If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back. I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever come) are light on details of the problem, and even lighter on details about their solutions. They seem to do a decent job of addressing most issues, because they rarely have recurring problems, but they never really talk about what they're doing.
    - antsar 11y ago
      Same experience here. I asked Support once about an ongoing outage and was told that "there were some network related issues". Really? I wouldn't have guessed...
- ryanlol 11y ago
  Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)
  - dasil003 11y ago
    I didn't downvote, but I can tell you almost certainly it was because of the snark.
  - scurvy 11y ago
    I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it. If you don't already have it enabled....well, good luck mate.
    - jsmthrowaway 11y ago
      As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.
      - scurvy 11y ago
        Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there. From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?
        X-Istence 11y ago
        Just because they purchase transit doesn't mean they don't have their own AS. Everyone has to purchase transit at some point. Transit != AS.
drzaiusapelord 11y ago
I've been with Linode for years and they've fought off more than a few DDOS attacks in the past, so they certainly are competent in that regard. I think this one must be a higher level of severity than previous ones. Generally, they seem ok with this stuff. I don't think there are any DDOS-proof VPS's out there.
kauegimenes 11y ago
This type of action is what the people behind the DDOS attack are looking for. Its not Linode fault...
- jedicoffee 11y ago
  Thank you for being so understanding!
  - electroly 11y ago
    Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. I filed a support ticket trying to get more information about the outage, and the response I got was absolutely worthless. No part of this has made me feel better about Linode at all. AWS is so massive that they can just sustain most DDOSes, and they write real postmortems after attacks. They're not as fast as Linode, but what good is speed if my site is completely vulnerable to every botnet that decides to look Linode's way?
    - workitout 11y ago
      >> Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. Then it will also be the same when you are at XYX ISP and they get DDOS'ed. This isn't a Linode specific problem. >> AWS is so massive that they can just sustain most DDOSes No. http://stackoverflow.com/questions/13244713/does-aws-protect-my-site-from-ddos-attacking http://stackoverflow.com/questions/13244713/does-aws-protect... "It will not, however, employ security algorithms needed to deflect application layer attacks (e.g., HTTP flood) nor will it help against larger network layer threats." https://www.cloudflare.com/ddos/ https://www.cloudflare.com/ddos/ "Layer 3 and 4 attacks are difficult—if not impossible—to mitigate with an on-premise solution. "

alexandrerond 11y ago

I am not sure there is a magic way to mitigate these attacks permanently other than playing cat and mouse until the attacker gets tired. Linode has in fact mitigated the attacks, only to find them pointing to a different part of their infra or DCs. At this point, it is probably a good sign that they haven't gone completely down (for long). Current tooling and botnets make DDoS-ing fairly easy today, despite countermeasures.

kyrra 11y ago

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

mmaunder 11y ago
That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing? Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.
- lftl 11y ago
  If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back. I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever come) are light on details of the problem, and even lighter on details about their solutions. They seem to do a decent job of addressing most issues, because they rarely have recurring problems, but they never really talk about what they're doing.
  - antsar 11y ago
    Same experience here. I asked Support once about an ongoing outage and was told that "there were some network related issues". Really? I wouldn't have guessed...
ryanlol 11y ago
Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)
- dasil003 11y ago
  I didn't downvote, but I can tell you almost certainly it was because of the snark.
- scurvy 11y ago
  I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it. If you don't already have it enabled....well, good luck mate.
  - jsmthrowaway 11y ago
    As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.
    - scurvy 11y ago
      Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there. From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?
      - X-Istence 11y ago
        Just because they purchase transit doesn't mean they don't have their own AS. Everyone has to purchase transit at some point. Transit != AS.

drzaiusapelord 11y ago

I've been with Linode for years and they've fought off more than a few DDOS attacks in the past, so they certainly are competent in that regard. I think this one must be a higher level of severity than previous ones. Generally, they seem ok with this stuff. I don't think there are any DDOS-proof VPS's out there.

kauegimenes 11y ago

This type of action is what the people behind the DDOS attack are looking for. Its not Linode fault...

jedicoffee 11y ago
Thank you for being so understanding!
- electroly 11y ago
  Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. I filed a support ticket trying to get more information about the outage, and the response I got was absolutely worthless. No part of this has made me feel better about Linode at all. AWS is so massive that they can just sustain most DDOSes, and they write real postmortems after attacks. They're not as fast as Linode, but what good is speed if my site is completely vulnerable to every botnet that decides to look Linode's way?
  - workitout 11y ago
    >> Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. Then it will also be the same when you are at XYX ISP and they get DDOS'ed. This isn't a Linode specific problem. >> AWS is so massive that they can just sustain most DDOSes No. http://stackoverflow.com/questions/13244713/does-aws-protect-my-site-from-ddos-attacking http://stackoverflow.com/questions/13244713/does-aws-protect... "It will not, however, employ security algorithms needed to deflect application layer attacks (e.g., HTTP flood) nor will it help against larger network layer threats." https://www.cloudflare.com/ddos/ https://www.cloudflare.com/ddos/ "Layer 3 and 4 attacks are difficult—if not impossible—to mitigate with an on-premise solution. "