7 ms·

DoS attacks that took down big game sites abused Web’s time-synch protocol

43 points by RougeFemme 13y ago

chris 13y ago
We (Weebly) had 18Gbps of UDP/123 (NTP) traffic sent our way on New Years eve -- definitely one of the larger attacks we've seen recently.
- midas007 13y ago
  This sound like what brought down the GnuPG main site. Sounds like an Anonymous(TM)-type thing.
nradov 13y ago
So both DNS and NTP can be abused in amplification DoS attacks. The obvious question is: which other protocols can potentially be exploited the same way?
- rwg 13y ago
  Any protocol that rides in UDP and that will reply to an unsolicited n byte packet with a >n byte response. (But if the amplification factor is close to 1, then you might as well have your botnet just attack the target directly.) Devices that speak SNMP and use "public" for their community could be used in an amplification attack. I don't know what the amplification factor would be, though, and I imagine there are loads more Internet-facing DNS and NTP servers than Internet-facing SNMP agents configured with a "public" community...
  - devicenull 13y ago
    SNMP is already being abused for this
    - midas007 13y ago
      SNMP has a history of being horribly insecure, because it was assumed people wouldn't put their mgmt hardware on the public IPs. It should only be deployed in trusted networks. It's about as secure as nonkrb5, unencrypted telnet.
- codexon 13y ago
  The most common attacks we have seen come from DNS, SNMP, and Chargen. You can get up to a 50k byte packet from a 64 byte response, but this is highly variable. SNMP is more prevalent than you might think, it is present and unsecured in many firewalls, routers, and printers.
  - midas007 13y ago
    It's funny you should mention this. So I used to be the lead sysadmin for the largest profit-center dept at one of the most well known universities. I left due to pressure to degrade security of the network for credit card processing and cash registers. It wasn't a protest as much as wanting to keep some appearance of integrity. On the plus side, the institution managed to deploy departmental firewalls and take Joe Sixpack's and Sally Sue's business desktops and servers off public IPs. If you wonder how many seconds it take to infect an unpatched windows box desktop, it's other the order of 60 s +- 25%. Yes, the desktop staff intentionally tried a few times just for kicks. It's a cracker's paradise.... fast links and there are like 2 semi-security people. But their duties and tasks are so diffuse, it's almost impossible to catch anything except boxes. Their organizational resistance even pushed away a couple of brilliant people you might have ran into at hope|ccc|bh.... Instituting a security officer and listening to them + letting them reach out to teach, influence are two different things.
ghshephard 13y ago
Easy DOS to block upstream, thankfully. Time seems like the sort of thing that a CoLo should provided to it's tenants. GPS time is likely good enough for 99% of sites, and, the final 1% who really require super reliable time, and need to be concerned about GPS spoofing/outages, can install a local atomic clock in their cabinet.
- sargun 13y ago
  How would you block it? Any packet with the source port of NTP should be dropped? No. ISPs should not get into the business of default block rules. This gets into all sorts of problems. At what point does a block become unreasonable? What if a tenant needs multiple sites in sync -- they must run NTP between them (or PTP) in order to determine their drift from one another. If you really want upstream blocking of traffic, the best thing to look towards is BGP Flowspec (RFC5575)
  - ghshephard 13y ago
    "How would you block it? " The same way all Upstream providers blocks a DDOS - by characterizing the DDOS flow, scrubbing, and dropping any packet which matches that pattern. The idea here, is that during a DDOS, where you might get a bit more NTP drops than normal, none of the local users should be impacted if they had a quality local time source. "What if a tenant needs multiple sites in sync" That's why you have a quality time source. By definition, they are "correct" to within any tolerance that all but CERN physics experiments care about, and they don't use NTP anyways. See: http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co... for an example of using synchronized time globally, without reliance on running NTP between sites. In particular, check out section 3, "TrueTime"
    - scurvy 13y ago
      It's trivial to add a UDP policer to ntp traffic to your ntp hosts. Also, how many people are exposing NTP externally on the same IP's they're running regular content (http/https) on? If you're allowing UDP/123 to your web servers, you need to fire yourself. I guess what I'm trying to say is, you should already have this ACL'd on your own.
skymt 13y ago
Why are ISPs letting spoofed traffic leave their network? I don't work at that level, but I thought this was a solved problem.
- codexon 13y ago
  1. Because it costs time and money. 2. It doesn't help yourself. So for many companies the cost can't be justified to the shareholders.
- sargun 13y ago
  I imagine you're suggesting most ISPs implement BCP38? A lot of organizations implement BCP38, but the overhead of BCP38 can actually be a lot, especially once you get far above the DSLAM / CMTS. Not only is it not free, it is not straightforward. In some more complex network architectures, you don't even know what the sources of downstream packets might be (case: downstream has two transits, and they don't want to bring ingress traffic via transit B, because transit B has ridiculous pricing).
  - cma 13y ago
    Its value is way higher than its overhead, but the overhead is paid by the guy who essentially doesn't benefit from it.
    - _delirium 13y ago
      I agree there's a payer/benefit mismatch, since filtering your own egress only protects other people's networks. But I would've thought it would still end up being incentivized somehow through things like peering or transit contracts. I don't have a good insight into how such contracts work, so I could be way off. My assumption was that in at least some such situations the other party would care if you're feeding them bogus stuff through the link, even if you don't care, so they would be motivated to require that you do egress filtering on your end as a condition of getting the link.
- devicenull 13y ago
  because it's profitable? there are providers that allow this because it gets them a lot of sales.
mirashii 13y ago
As someone who was considering Black Lotus as a potential service for DDOS protection, they had a number of failures during this whole episode that make it hard to take them quite seriously. First, and most importantly, they suffered many hours of downtime to many sites hosted on their service, including their own web page, and still to this day have not tweeted or given any public statement about the incident. A lot of Googling got me some responses to customers who complained in private with weak justification about it being their upstream provider's problem and not theirs. The fact that they now estimate that it was 28GBit/s in traffic makes me even less trusting of their service, given that much larger DDOSes are flying around. If anyone has any other services that are competitive, other than Cloudflare, I'd be interested in hearing about them out of curiosity. It seems there should be more than one company that can do this well.
- scurvy 13y ago
  defense.net and Prolexic come to mind before Black Lotus or CloudFlare. Black Lotus and CloudFlare are bottom of the bucket options compared to defense and Prolexic. Keep in mind that Prolexic was recently acquired by Akamai, so prices will skyrocket soon.
- mjn 13y ago
  It only works for certain kinds of sites, but if it's a site you can host on third-party hosting (as opposed to needing DDOS protection of your own network or VPS), Nearlyfreespeech.net has a pretty good track record with mitigating DDOS on sites they host. They also have a strong ethos of not blaming the customer for "provoking" the attack, or terminating them for the inconvenience.
- pktgen 13y ago
  That's how Black Lotus does business. It's like the time when they were caught spamming and were made to look like complete fools on WHT.
PaulRobinson 13y ago
The Web != The Internet. NTP is not "the Web's time protocol". Really fed up with tech journalism these days assuming that nobody has quite finished the first few chapters of the "Networking for Dummies" book. Massive gap in the market for informed quality tech journalism.
p1mrx 13y ago
The Internet is not the Web, and NTP has nothing to do with the latter.

43 points by RougeFemme 13y ago

chris 13y ago

We (Weebly) had 18Gbps of UDP/123 (NTP) traffic sent our way on New Years eve -- definitely one of the larger attacks we've seen recently.

midas007 13y ago
This sound like what brought down the GnuPG main site. Sounds like an Anonymous(TM)-type thing.

nradov 13y ago

So both DNS and NTP can be abused in amplification DoS attacks. The obvious question is: which other protocols can potentially be exploited the same way?

rwg 13y ago
Any protocol that rides in UDP and that will reply to an unsolicited n byte packet with a >n byte response. (But if the amplification factor is close to 1, then you might as well have your botnet just attack the target directly.) Devices that speak SNMP and use "public" for their community could be used in an amplification attack. I don't know what the amplification factor would be, though, and I imagine there are loads more Internet-facing DNS and NTP servers than Internet-facing SNMP agents configured with a "public" community...
- devicenull 13y ago
  SNMP is already being abused for this
  - midas007 13y ago
    SNMP has a history of being horribly insecure, because it was assumed people wouldn't put their mgmt hardware on the public IPs. It should only be deployed in trusted networks. It's about as secure as nonkrb5, unencrypted telnet.
codexon 13y ago
The most common attacks we have seen come from DNS, SNMP, and Chargen. You can get up to a 50k byte packet from a 64 byte response, but this is highly variable. SNMP is more prevalent than you might think, it is present and unsecured in many firewalls, routers, and printers.
- midas007 13y ago
  It's funny you should mention this. So I used to be the lead sysadmin for the largest profit-center dept at one of the most well known universities. I left due to pressure to degrade security of the network for credit card processing and cash registers. It wasn't a protest as much as wanting to keep some appearance of integrity. On the plus side, the institution managed to deploy departmental firewalls and take Joe Sixpack's and Sally Sue's business desktops and servers off public IPs. If you wonder how many seconds it take to infect an unpatched windows box desktop, it's other the order of 60 s +- 25%. Yes, the desktop staff intentionally tried a few times just for kicks. It's a cracker's paradise.... fast links and there are like 2 semi-security people. But their duties and tasks are so diffuse, it's almost impossible to catch anything except boxes. Their organizational resistance even pushed away a couple of brilliant people you might have ran into at hope|ccc|bh.... Instituting a security officer and listening to them + letting them reach out to teach, influence are two different things.

ghshephard 13y ago

Easy DOS to block upstream, thankfully. Time seems like the sort of thing that a CoLo should provided to it's tenants. GPS time is likely good enough for 99% of sites, and, the final 1% who really require super reliable time, and need to be concerned about GPS spoofing/outages, can install a local atomic clock in their cabinet.

sargun 13y ago
How would you block it? Any packet with the source port of NTP should be dropped? No. ISPs should not get into the business of default block rules. This gets into all sorts of problems. At what point does a block become unreasonable? What if a tenant needs multiple sites in sync -- they must run NTP between them (or PTP) in order to determine their drift from one another. If you really want upstream blocking of traffic, the best thing to look towards is BGP Flowspec (RFC5575)
- ghshephard 13y ago
  "How would you block it? " The same way all Upstream providers blocks a DDOS - by characterizing the DDOS flow, scrubbing, and dropping any packet which matches that pattern. The idea here, is that during a DDOS, where you might get a bit more NTP drops than normal, none of the local users should be impacted if they had a quality local time source. "What if a tenant needs multiple sites in sync" That's why you have a quality time source. By definition, they are "correct" to within any tolerance that all but CERN physics experiments care about, and they don't use NTP anyways. See: http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co... for an example of using synchronized time globally, without reliance on running NTP between sites. In particular, check out section 3, "TrueTime"
  - scurvy 13y ago
    It's trivial to add a UDP policer to ntp traffic to your ntp hosts. Also, how many people are exposing NTP externally on the same IP's they're running regular content (http/https) on? If you're allowing UDP/123 to your web servers, you need to fire yourself. I guess what I'm trying to say is, you should already have this ACL'd on your own.

skymt 13y ago

Why are ISPs letting spoofed traffic leave their network? I don't work at that level, but I thought this was a solved problem.

codexon 13y ago
1. Because it costs time and money. 2. It doesn't help yourself. So for many companies the cost can't be justified to the shareholders.
sargun 13y ago
I imagine you're suggesting most ISPs implement BCP38? A lot of organizations implement BCP38, but the overhead of BCP38 can actually be a lot, especially once you get far above the DSLAM / CMTS. Not only is it not free, it is not straightforward. In some more complex network architectures, you don't even know what the sources of downstream packets might be (case: downstream has two transits, and they don't want to bring ingress traffic via transit B, because transit B has ridiculous pricing).
- cma 13y ago
  Its value is way higher than its overhead, but the overhead is paid by the guy who essentially doesn't benefit from it.
  - _delirium 13y ago
    I agree there's a payer/benefit mismatch, since filtering your own egress only protects other people's networks. But I would've thought it would still end up being incentivized somehow through things like peering or transit contracts. I don't have a good insight into how such contracts work, so I could be way off. My assumption was that in at least some such situations the other party would care if you're feeding them bogus stuff through the link, even if you don't care, so they would be motivated to require that you do egress filtering on your end as a condition of getting the link.
devicenull 13y ago
because it's profitable? there are providers that allow this because it gets them a lot of sales.

mirashii 13y ago

As someone who was considering Black Lotus as a potential service for DDOS protection, they had a number of failures during this whole episode that make it hard to take them quite seriously. First, and most importantly, they suffered many hours of downtime to many sites hosted on their service, including their own web page, and still to this day have not tweeted or given any public statement about the incident. A lot of Googling got me some responses to customers who complained in private with weak justification about it being their upstream provider's problem and not theirs. The fact that they now estimate that it was 28GBit/s in traffic makes me even less trusting of their service, given that much larger DDOSes are flying around. If anyone has any other services that are competitive, other than Cloudflare, I'd be interested in hearing about them out of curiosity. It seems there should be more than one company that can do this well.

scurvy 13y ago
defense.net and Prolexic come to mind before Black Lotus or CloudFlare. Black Lotus and CloudFlare are bottom of the bucket options compared to defense and Prolexic. Keep in mind that Prolexic was recently acquired by Akamai, so prices will skyrocket soon.
mjn 13y ago
It only works for certain kinds of sites, but if it's a site you can host on third-party hosting (as opposed to needing DDOS protection of your own network or VPS), Nearlyfreespeech.net has a pretty good track record with mitigating DDOS on sites they host. They also have a strong ethos of not blaming the customer for "provoking" the attack, or terminating them for the inconvenience.
pktgen 13y ago
That's how Black Lotus does business. It's like the time when they were caught spamming and were made to look like complete fools on WHT.

PaulRobinson 13y ago

The Web != The Internet. NTP is not "the Web's time protocol". Really fed up with tech journalism these days assuming that nobody has quite finished the first few chapters of the "Networking for Dummies" book. Massive gap in the market for informed quality tech journalism.

p1mrx 13y ago

The Internet is not the Web, and NTP has nothing to do with the latter.