5 ms·

Ask HN: How to deal with SYN FLOOD and extortion

40 points by xal 18y ago

We made a big mistake in our server setup. We bought beefy as app and db servers but went for ultra reliable sun ultrasparks and openbsd for firewalls. This was fine until the DDOS started. After 3 days of many different attacks we now how much better firewall hardware installed thanks to our fantastic colocation which went the extra mile and helped us when we needed their help most but we are now surviving on total brute force. Our firewalls are faster then the SYN Floods that hit us. This is an arms race that we cannot win in the long run. Yes we can buy more hardware but it's much easier to infect more machines with bots over time. How do people protect themselves against extortion and malicious ddos attacks? What software / hardware protects the bigger sites on the net?

Brushfire 18y ago
Move yourself into the cloud and let someone else deal with it. Seriously. I'm a big fan of outsourcing anything you arent 'great' at. In this case, its running firewalls. Unless of course your app is something related to server maintenance or firewalls or security, etc. Most software companies really shouldnt try to deal with this stuff -- I had to learn the hard way. You should only spend your time on your applications, not anything else that doesnt directly benefit you. You can get lost in the maintenance.
- Tichy 18y ago
  Just curious, but how would "the cloud" help with such a problem? Sure, supposedly you could summon infinite computing power to handle the DDOS requests. But the computing power would not come for free, so the DDOS would still do serious financial damage.
  - Brushfire 18y ago
    You are paying either way. Thats just the way it works. However, under one option, you pay and Amazon deals with the actual load, downtime, server stress, hardware improvements, etc. Under the other, you get to deal with all of that AND get to be pissed off AND accomplish nothing for your business at the same time.
    - xal 18y ago
      I'm curious, anyone has experience dealing with DDOS attacks on EC2? It does seem like a good option...
      - wmf 18y ago
        If you're using an instance as a load balancer it's limited to 1 Gbps.
  - irrelative 18y ago
    There are several companies that create DDoS mitigation hardware/software. I have good knowledge that Amazon (et al) use several of these companies for solutions.
tptacek 18y ago
Any large ISP should have a process and gear in house to mitigate DDOS attacks; you should escalate, first to your hosting provider and then to their upstream provider. Every ISP does something a little bit different for large-scale attacks. Some of them off-ramp traffic to scrubbers, some of them have inline devices that can characterize and block SYN floods. My previous employer, Arbor Networks, now has gear deployed at something like 90% of the worldwide tier 1 and tier 2 ISPs that will detect any major flooding event and generate a report which might be helpful to you. What I'd watch out for is the dozen odd fly-by-night operations that are promising you that, for a monthly fee or a one-time purchase of some $100k box, you can block these attacks on your own network.
- lsc 18y ago
  I'd extend that warning... watch out for any company, even if it's large and reputable and the salesmen have nice suits, that promises to solve your DoS problem by selling you an expensive firewall. If your traffic is below 100Mbps, there's nothing the fancy box can do that your OpenBSD box can't. (this changes as your traffic climbs significantly above 100Mbps, but most of us little guys are below that line.)
  - iuguy 18y ago
    I'd go a bit further and say that even over 100Mbps you're not going to get much protection over an OpenBSD box or a Sun system running a decent Firewall. The key thing is to try to block it upstream. Your firewall (and indeed anyone's firewall) is only really designed to restrict IP-layer access to specific source and destination hosts, ports and datagram types. Unless it was built by Cisco, in which case it was probably designed to do a whole load of other stuff, none of which will help you right now. There are limited things you can do to the IP stacks of most OSes these days that can help discard unsolicited packets, and reduce resource consumption on the local boxes (if that's high) but an upstream solution is likely to be more optimal in most cases.
lsc 18y ago
your provider (or your network guy, possibly you) needs to blackhole the source of these attacks (preferably notifying the ISPs in question.) most of the better co-location providers will give you a way to blackhole traffic from certain IPs. Really, you should be able to do that from your OpenBSD firewalls without help from anyone else, assuming the problem is too many syn packets rather than too much traffic. dropping all packets from a particular source IP is not expensive, even when you multiply that by a large number of source IPs. <please ignore this comment, I'm completely wrong. see the next comment.>
- asmithmd1 18y ago
  Black listing the source IP's is NOT going to help. The attacker can just spoof any IP, he doesn't care about actually connecting he just wants to tie-up your server http://en.wikipedia.org/wiki/SYN_flood http://en.wikipedia.org/wiki/SYN_flood
  - axod 18y ago
    As far as I'm aware (Correct me if I'm wrong anyone in the know), the process for any spoofed source address flood would be: * Determine which upstream provider it's coming from * Contact that provider, and ask them to investigate * They do the same thing * If a provider finds out the source is local, then they block it / contact the person(s) responsible etc Of course it gets harder if it's a botnet or some other system that is distributed over a large number of sources.
    - blhack 18y ago
      I think that, if we're talking about the level of traffic that I think we're talking about here, it would almost HAVE to be a botnet. either that, or somebody at one of the bandwidth providers got really <i>really</i> pissed off at you...if the later is the case the guy (or girl...they can be vindictive as hell) has probably already been fired since, you know, they sortof monitor these sorts of things.
      - 18y ago
axod 18y ago
I've had a few DDOS attacks in the past, first when hosting with slicehost, and then when at linode. Running IRC related stuff, these attacks are pretty much expected from time to time. Both dealt with them fantastically, keeping me updated on progress etc. As far as I remember, there was only one instance when they decided (slicehost) to take my server 'offline' and just wait for it to pass/upstream providers to deal etc. The other times they were quickly able to escalate to upstream providers with minimal impact. I've had a few at linode, and whilst it does use up some bandwidth allowance, it hasn't been a big deal at all. Obviously if you're able to, get more IPs, use a few data centers, etc. distribute yourself so that an attack on any one part doesn't have a big impact.
patrickg-zill 18y ago
If they are never connecting, then why not set up a SYN "honeypot"? As soon as a valid user connects, redirect them to a different site or IP address that you do not advertise, that actually hosts your app and db servers. Thus you could rent or colo a small system on a different network, maybe even in a different part of the country and then redirect valid users to an entirely different location with a cookie in the URL. No cookie no service.
- moe 18y ago
  The attacker would simply move the syn-flood to where the candy is.
mpk 18y ago
The big boys seem to be a little reluctant to publish information on their DDoS strategies. (If anyone has information that says otherwise, it's much appreciated). I spent some time googling around and in case anyone's interested, these links seem to cover most of the mitigation patterns. A slideshow covering common mitigation techniques, http://www.slideshare.net/intruguard/10-ddos-mitigation-techniques-presentation http://www.slideshare.net/intruguard/10-ddos-mitigation-tech... A Cisco whitepaper (which, as usual, has en emphasis on Cisco kit and a long URL), http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps6264/ps5888/prod_white_paper0900aecd8011e927_ns615_Networking_Solutions_White_Paper.html http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps... A fairly comprehensive PDF document - A Survey of Active and Passive Defence Mechanisms against DDoS Attacks http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921 http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921
- lacker 18y ago
  The big boys seem to be a little reluctant to publish information on their DDoS strategies. Makes sense, the bad guys are just as likely (if not more) to read up about them.
- tptacek 18y ago
  I don't know how secret this stuff is; I think if you poke around NANOG, you'll find a lot of material. If there's something that makes DDoS mitigation mysterious, it's how ad hoc all this stuff is. What I've had firsthand experience with is: * Tier-1's profile all their traffic, either directly or with flow export, and will get alerts if they see spikes to certain netblocks, or spikes to specific /32s with specific characteristics. * Inside a Tier-1, the third-tier support people usually have additional monitoring they can enable for customers if an incident has been escalated to them. They get hundreds of these calls a week. * A lot of Tier-1's can quickly reroute traffic to a specific /32 to run through special-purpose filtering setups, which might be a DDoS box like a Cisco/Riverhead, or an IPS like TippingPoint, or even just a Cat with lots of TCAM space set aside for filtering. * A lot of Tier-1's --- maybe all at this point --- have some mechanism set up to share signatures of attacks so they can push filtering further upstream. When I watched that stuff happening, the sense I got of it was that this was really reserved for things like global botnet C&C. What I'd add to the discussion on products is, most of what's built to combat DDoS is really only useful if you sell transit. If you're a Fortune 500, I know there are ISPs where you can get Cisco or Arbor gear deployed on the head end specially for you. But that's a Fortune 500, not 37 Signals.
datums 18y ago
If they have already attempted to extort $$. You should keep the mitigation device in place. There are a few devices that do packet inspection and can handle multi gigabyte size attacks.But like many have mentioned contact the upstream provider

40 points by xal 18y ago

Brushfire 18y ago

Move yourself into the cloud and let someone else deal with it. Seriously. I'm a big fan of outsourcing anything you arent 'great' at. In this case, its running firewalls. Unless of course your app is something related to server maintenance or firewalls or security, etc. Most software companies really shouldnt try to deal with this stuff -- I had to learn the hard way. You should only spend your time on your applications, not anything else that doesnt directly benefit you. You can get lost in the maintenance.

Tichy 18y ago
Just curious, but how would "the cloud" help with such a problem? Sure, supposedly you could summon infinite computing power to handle the DDOS requests. But the computing power would not come for free, so the DDOS would still do serious financial damage.
- Brushfire 18y ago
  You are paying either way. Thats just the way it works. However, under one option, you pay and Amazon deals with the actual load, downtime, server stress, hardware improvements, etc. Under the other, you get to deal with all of that AND get to be pissed off AND accomplish nothing for your business at the same time.
  - xal 18y ago
    I'm curious, anyone has experience dealing with DDOS attacks on EC2? It does seem like a good option...
    - wmf 18y ago
      If you're using an instance as a load balancer it's limited to 1 Gbps.
- irrelative 18y ago
  There are several companies that create DDoS mitigation hardware/software. I have good knowledge that Amazon (et al) use several of these companies for solutions.

tptacek 18y ago

Any large ISP should have a process and gear in house to mitigate DDOS attacks; you should escalate, first to your hosting provider and then to their upstream provider. Every ISP does something a little bit different for large-scale attacks. Some of them off-ramp traffic to scrubbers, some of them have inline devices that can characterize and block SYN floods. My previous employer, Arbor Networks, now has gear deployed at something like 90% of the worldwide tier 1 and tier 2 ISPs that will detect any major flooding event and generate a report which might be helpful to you. What I'd watch out for is the dozen odd fly-by-night operations that are promising you that, for a monthly fee or a one-time purchase of some $100k box, you can block these attacks on your own network.

lsc 18y ago
I'd extend that warning... watch out for any company, even if it's large and reputable and the salesmen have nice suits, that promises to solve your DoS problem by selling you an expensive firewall. If your traffic is below 100Mbps, there's nothing the fancy box can do that your OpenBSD box can't. (this changes as your traffic climbs significantly above 100Mbps, but most of us little guys are below that line.)
- iuguy 18y ago
  I'd go a bit further and say that even over 100Mbps you're not going to get much protection over an OpenBSD box or a Sun system running a decent Firewall. The key thing is to try to block it upstream. Your firewall (and indeed anyone's firewall) is only really designed to restrict IP-layer access to specific source and destination hosts, ports and datagram types. Unless it was built by Cisco, in which case it was probably designed to do a whole load of other stuff, none of which will help you right now. There are limited things you can do to the IP stacks of most OSes these days that can help discard unsolicited packets, and reduce resource consumption on the local boxes (if that's high) but an upstream solution is likely to be more optimal in most cases.

lsc 18y ago

your provider (or your network guy, possibly you) needs to blackhole the source of these attacks (preferably notifying the ISPs in question.) most of the better co-location providers will give you a way to blackhole traffic from certain IPs. Really, you should be able to do that from your OpenBSD firewalls without help from anyone else, assuming the problem is too many syn packets rather than too much traffic. dropping all packets from a particular source IP is not expensive, even when you multiply that by a large number of source IPs. <please ignore this comment, I'm completely wrong. see the next comment.>

asmithmd1 18y ago
Black listing the source IP's is NOT going to help. The attacker can just spoof any IP, he doesn't care about actually connecting he just wants to tie-up your server http://en.wikipedia.org/wiki/SYN_flood http://en.wikipedia.org/wiki/SYN_flood
- axod 18y ago
  As far as I'm aware (Correct me if I'm wrong anyone in the know), the process for any spoofed source address flood would be: * Determine which upstream provider it's coming from * Contact that provider, and ask them to investigate * They do the same thing * If a provider finds out the source is local, then they block it / contact the person(s) responsible etc Of course it gets harder if it's a botnet or some other system that is distributed over a large number of sources.
  - blhack 18y ago
    I think that, if we're talking about the level of traffic that I think we're talking about here, it would almost HAVE to be a botnet. either that, or somebody at one of the bandwidth providers got really <i>really</i> pissed off at you...if the later is the case the guy (or girl...they can be vindictive as hell) has probably already been fired since, you know, they sortof monitor these sorts of things.
    - 18y ago

axod 18y ago

I've had a few DDOS attacks in the past, first when hosting with slicehost, and then when at linode. Running IRC related stuff, these attacks are pretty much expected from time to time. Both dealt with them fantastically, keeping me updated on progress etc. As far as I remember, there was only one instance when they decided (slicehost) to take my server 'offline' and just wait for it to pass/upstream providers to deal etc. The other times they were quickly able to escalate to upstream providers with minimal impact. I've had a few at linode, and whilst it does use up some bandwidth allowance, it hasn't been a big deal at all. Obviously if you're able to, get more IPs, use a few data centers, etc. distribute yourself so that an attack on any one part doesn't have a big impact.

patrickg-zill 18y ago

If they are never connecting, then why not set up a SYN "honeypot"? As soon as a valid user connects, redirect them to a different site or IP address that you do not advertise, that actually hosts your app and db servers. Thus you could rent or colo a small system on a different network, maybe even in a different part of the country and then redirect valid users to an entirely different location with a cookie in the URL. No cookie no service.

moe 18y ago
The attacker would simply move the syn-flood to where the candy is.

mpk 18y ago

The big boys seem to be a little reluctant to publish information on their DDoS strategies. (If anyone has information that says otherwise, it's much appreciated). I spent some time googling around and in case anyone's interested, these links seem to cover most of the mitigation patterns. A slideshow covering common mitigation techniques, http://www.slideshare.net/intruguard/10-ddos-mitigation-techniques-presentation http://www.slideshare.net/intruguard/10-ddos-mitigation-tech... A Cisco whitepaper (which, as usual, has en emphasis on Cisco kit and a long URL), http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps6264/ps5888/prod_white_paper0900aecd8011e927_ns615_Networking_Solutions_White_Paper.html http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps... A fairly comprehensive PDF document - A Survey of Active and Passive Defence Mechanisms against DDoS Attacks http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921 http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921

lacker 18y ago
The big boys seem to be a little reluctant to publish information on their DDoS strategies. Makes sense, the bad guys are just as likely (if not more) to read up about them.
tptacek 18y ago
I don't know how secret this stuff is; I think if you poke around NANOG, you'll find a lot of material. If there's something that makes DDoS mitigation mysterious, it's how ad hoc all this stuff is. What I've had firsthand experience with is: * Tier-1's profile all their traffic, either directly or with flow export, and will get alerts if they see spikes to certain netblocks, or spikes to specific /32s with specific characteristics. * Inside a Tier-1, the third-tier support people usually have additional monitoring they can enable for customers if an incident has been escalated to them. They get hundreds of these calls a week. * A lot of Tier-1's can quickly reroute traffic to a specific /32 to run through special-purpose filtering setups, which might be a DDoS box like a Cisco/Riverhead, or an IPS like TippingPoint, or even just a Cat with lots of TCAM space set aside for filtering. * A lot of Tier-1's --- maybe all at this point --- have some mechanism set up to share signatures of attacks so they can push filtering further upstream. When I watched that stuff happening, the sense I got of it was that this was really reserved for things like global botnet C&C. What I'd add to the discussion on products is, most of what's built to combat DDoS is really only useful if you sell transit. If you're a Fortune 500, I know there are ISPs where you can get Cisco or Arbor gear deployed on the head end specially for you. But that's a Fortune 500, not 37 Signals.

datums 18y ago

If they have already attempted to extort $$. You should keep the mitigation device in place. There are a few devices that do packet inspection and can handle multi gigabyte size attacks.But like many have mentioned contact the upstream provider