3 ms·

[Disclaimer: Linode employee] Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey,

[Disclaimer: Linode employee] Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.

atmosx 11y ago
Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?
- alexforster 11y ago
  Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks. We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.
  - jgord 11y ago
    As a happy linode user, your update is much appreciated. I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires. I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.
    - nickpsecurity 11y ago
      Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm: https://protonmail.com/blog/ddos-protection-guide/ https://protonmail.com/blog/ddos-protection-guide/
jonknee 11y ago
I think "whole data center is offline" is the sort of thing you call the boss for. Otherwise why is there a boss at all?

atmosx 11y ago

Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?

alexforster 11y ago
Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks. We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.
- jgord 11y ago
  As a happy linode user, your update is much appreciated. I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires. I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.
  - nickpsecurity 11y ago
    Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm: https://protonmail.com/blog/ddos-protection-guide/ https://protonmail.com/blog/ddos-protection-guide/

jonknee 11y ago

I think "whole data center is offline" is the sort of thing you call the boss for. Otherwise why is there a boss at all?