7 ms·

DNSMadeEasy under major (over 50Gbps) DDoS Attack out of China

110 points by KrisJordan 16y ago

makmanalp 16y ago
I'm interning at a major managed DNS provider and this actually happens more often than you think. Most of the time it ends by working with telcos like level3, cogent etc, blackholing routes, sacrificing a certain geographical location in favor of keeping others alive and basically just waiting it out. It's interesting how because of the nature of the internet, DDoSes are hard to deal with.
- drivingmenuts 16y ago
  Too bad we can't just permanently sacrifice China.
  - dnsworks 16y ago
    There's an anti-DDOS service out there, Progent or something. Part of their solution adds all of APNIC's ip addresses into your router's bogon list when you're under attack. It turns out for most DDOS's if you turn off china, the attack just ends.
    - tptacek 16y ago
      I think you're thinking of Prolexic, and my recommendation is that you take any of their claims with a heaping helping of salt. Things may have changed in the 5 years since I've totally forgotten everything I knew about DoS mitigation, but it wasn't the case then that you could simply black hole China to evade them. A more common mechanism used by real networks to mitigate these attacks in production is to pinpoint the target of the attack and offramp its traffic in the ISP core to a "regional scrubbing center" where advanced filtering tools (for instance, packet filters that can handle ACLs with high tens of thousands of terms) can try to sort through the crap.
      - jacquesm 16y ago
        At my colo that is exactly the sort of strategy they use, they have bought a bunch of routers that use FPGA based filters for that. The one scenario I can think of where that might be a problem is if they'd start flooding all the known hosts in a network for the specific purpose of overwhelming the routers. And even those hardware based filtering tools have upper limits.
        dnsworks 16y ago
        There's really no need to do that. A large enough BotNet can take out almost any host doing nothing more than connecting and doing a GET / HTTP/1.1. Few websites can handle a sudden surge in traffic from 100,000 bots.
        jacquesm 16y ago
        The idea behind an ACL for such an attack is that the same hosts will be used over and over again, taken from a large pool of zombies. So, let's take your example of 100,000 bots, an ACL in the upstream router (as seen from the host) could be used to checked against to identify those packets from the zombies and discard them. After all, a single attack of all 100,000 bots at once will just bring the host to its knees for the time-out of the connections and then it will bounce back up again. So, to increase the effectiveness of the attack they reconnect after every lost connection asking for another resource. If they're smart they'll vary agent strings and other characteristics to make it hard to narrow down who is and who is not legit. Initially you don't have much to go on during such an attack and packet filtering is a reasonably expensive operation when you want to do it for a large number of hosts. So the strategy is to route all the traffic destined for that particular host through a router that has ACLs that are large enough to hold the total IP list for the botnet that is attacking the host, as these IPs become identified. You don't want to route all your traffic through there because then you'd have to do the relatively expensive filtering on all of the packets, even those not destined for that particular host. Now if an attacker were targeting the hosting facility they could thwart this strategy by sending requests to a larger number of hosts in the network in order to make life much harder for the crew fighting the attack. After all, you can't partition the problem anymore in to a portion that is targeted to the host and 'normal' traffic, effectively all the traffic could be bot traffic or it could be normal traffic, for all the receiving hosts. To be able to partition the problem into a smaller one where you can let say 90% or more of the traffic through unfiltered and only concentrate on the remaining 10% would make solving it a bit easier. On the other hand if the attackers are silly enough to re-use the same bots to attack different hosts they've actually given you a clue as to which IPs are bots. I hope that makes sense :)
  - vkdelta 16y ago
    If we want, we can :)
  - mrtron 16y ago
    It is easy to say that now - would you have said that about US cable modem accounts 10 years ago?
- tptacek 16y ago
  From 2001-2003 I led development on Arbor Peakflow DoS, which (when I left, at least) was far and away the most popular tool tier-1 ISPs used to manage these kinds of attacks, and my the experience I can relate is: * These attacks happen with shocking frequency (the major incentive ISPs have to mitigate them is not angry customers, but rather the drag it creates on their third-tier engineering staff to hunt down DDoS sources and craft ACLs for them) * There is actually not a whole hell of a lot you can do about them, even if you light up a whole tier 1 core network with mitigation tools. At the end of the day, a well-crafted DDoS attack looks pretty much identical to normal traffic; if you can black-hole China to defeat an attack, it wasn't very cluefully done.
  - makmanalp 16y ago
    Regarding your statement that it wasn't very cluefully done, isn't that sort of a hit and miss area? You can't really control much where your malware spreads (and thus where your bots are). As a sidenote, I've been reading your posts for a while now and it seems you always have something insightful to say - cheers!
    - peterwwillis 16y ago
      it's not difficult to control where malware spreads. directly attacking specific ranges is one easy method, as is coding the limitation into your bot or only sending commands to the ones in a certain range.
leftnode 16y ago
No wonder most of my sites were down or not functioning very well this morning. Hope they get everything operational soon, DNSMadeEasy is a great provider.
deleted 16y ago
[deleted]
cliffchang 16y ago
Can anyone give an idea of how large, compared to other DDOS attacks, 50 GBPS is?
- dnsworks 16y ago
  50gbps is pretty massive. Clearly somebody did something horrifically insulting to China like mention that they could maybe possible consider treating Tibet a little bit more humanely and stop turning it into one big whorehouse.. make a statement like that, and all of china ddos's you.
  - dogas 16y ago
    Sarcasm aside, is that really what motivates people in china to regularly pull off attacks like this?
    - paulitex 16y ago
      No. (having lived in China for multiple years) Pretty much all the political issues the West thinks of WRT China are not even given a cursory thought by most Chinese. China just isn't very political. Taiwan is basically a non-issue. Tibet is completely a non-issue. More to the topic: It's much more likely that the motivations are financial.
      - dnsworks 16y ago
        And who do you think pays these people to ddos sites like slideshare and posterous? (both of whom have been my customers during massive ddos's from China). I'm sure there's like one posting somewhere on Posterous that someone in the Chinese government didn't like, so they paid their team of script kiddies for time on their botnets to bully Posterous around. The same thing happened to Slideshare a few years ago. It's about China bullying people around and trying to censor the internet.
        jhancock 16y ago
wehriam 16y ago
What motivates an attack like this?
- Steve0 16y ago
  Probably some motive against one of the customers. If you can extort some cash out of a gambling site with short dns-ttl it could be worth your effort. Maybe the target's data center is more ddos-proof than the one from easydns, extortionists go for the weakest link.
  - mrtron 16y ago
    Back in the olden days of IRC, there was times when entire regions of the internet were taken offline resulting from a ddos for personal vendettas. For exactly that reason too - they attack the weakest link and often it isn't the host directly. In this case - it really could be anything. The cost of one of these attacks is next to zero. Rarely will the botnet owner lose any machines resulting from an attack. The unfortunate thing after one of these attacks is you have no way of preventing it or going after the source.
quizbiz 16y ago
Can someone please explain to me what DNSMadeEasy actually does? Is a domain provider like 1and1 a client of theirs or is my hosting provider a client? This entry on Hacker News led me to wikipedia where I went from an article about Level3 to an article about Tier 1 Network to Internet Backbone. I feel like I'm on the verge of understanding more about how the internet works but it's all a bit above my head.
- geocar 16y ago
  They're an anycast dns provider- basically the only kind of third-party DNS service that's worth anything at all. They have some neat domains-related tools that hook into their infrastructure that's kindof a pain to do yourself (even if you've got your own anycast-capable network)
FooBarWidget 16y ago
I'm wondering what the implications of DDOS are for website owners. What if you're on EC2 for example, will you be charged for the 300 TB of traffic? If so that would be an easy way to bankrupt a startup.
- dennisgorelik 16y ago
  The implications for website owners are that their web sites were temporary unavailable. Traffic simply does not reach the web site. Even normal traffic. For example, you type in your browser www.mywebsite.com. DNSMadeEasy would normally resolve it to your IP address (e.g. 111.222.123.12). But because of the DDoS attach -- mywebsite.com cannot be resolved into any IP address and you cannot open www.mywebsite.com at all.
  - dan_manges 16y ago
    That's the implication is there is a DDoS attack on your DNS provider, but I think FooBarWidget was inquiring about the implications of a DDoS attack on your website, and specifically if you would be charged for the bandwidth consumed by the attack.
    - jacquesm 16y ago
      You can - but not everywhere - negotiate that the service you pay for includes protection against DDOS attacks and that it's up to your provider to protect you. You'll pay a larger fee per mbit because they'll need to do more work for you in case you get hit but it might be worth it.
- peterwwillis 16y ago
  get akamized. you can also build traffic limits into your web stack or OS or network gear if you're afraid of traffic ramps hurting your wallet. like tptacek said, a good DDOS looks like normal traffic, so this could happen if you got slashdotted by 10 different news sites.
- nhnifong 16y ago
  My policy for this is to simply lock the doors and hide for an hour if my network traffic averages over 2 Mbps for 5 minutes. I would also send myself an email. So far this has never happened.
ximeng 16y ago
(Brief) discussion on webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=970837&highlight=china http://www.webhostingtalk.com/showthread.php?t=970837&hi... Another 30 gbps attack, discussion on webhostingtalk.com: http://www.webhostingtalk.com/showthread.php?t=966658&highlight=china http://www.webhostingtalk.com/showthread.php?t=966658&hi... Edit: interesting thread, apparently the reason it's 30gbps is that it's maxing out the link from China telecom, so legitimate Chinese customers are getting traffic dropped. FBI are involved. Suggested solution by some is to get traffic routed over more intercontinental links possibly via peering agreement with China Telecom, and beyond that political pressure.
petercooper 16y ago
I wonder if, long term, DDOS attacks will be to net neutrality as spam was to nice-and-easy e-mail setups. Now ISPs are running spam blacklists, blocking entire other ISPs at times, and it's a nightmare to run your own SMTP server and deliver mail reliably without jumping through hoops. If DDOS attacks become more and more annoying, they could be used as an excuse to violate net neutrality.
- SoftwareMaven 16y ago
  I'm not clear on how that would work. A good DDoS attack is nearly indistinguishable from normal traffic. How would net neutrality play into it?
  - petercooper 16y ago
    Really good spam e-mail is nearly indistinguishable from normal e-mail. Crazy layers of whitelisting/blacklisting/new DNS settings/laws/software and policies have been piled on top of the mail system to reduce the nefarious effects of spam sent en masse. While spam and DDOS attacks aren't directly comparable in terms of what they are, I'm speculating in terms of what negative effects continued and escalating DDOS attacks could have again in terms of laws, policies, white/blacklisting of entire networks/countries, and so forth.
est 16y ago
The GFW of China has a known vulnerability to amplify a UDP traffic to 2x to 3x as much.
meric 16y ago
I wonder if they're going to change the text on their website: "A DNS service with a 99.9999% uptime history is just the start! DNS Made Easy is so confident of it's uptime record that we offer the best service level agreement in the industry. That is why all businesses that require stable DNS decide to use DNS Made Easy. We have an industry leading 100% uptime gaurantee and will credit all accounts 500% of the downtime." Maybe its a rival DNS provider behind the attack?

110 points by KrisJordan 16y ago

I'm interning at a major managed DNS provider and this actually happens more often than you think. Most of the time it ends by working with telcos like level3, cogent etc, blackholing routes, sacrificing a certain geographical location in favor of keeping others alive and basically just waiting it out. It's interesting how because of the nature of the internet, DDoSes are hard to deal with.

drivingmenuts 16y ago
Too bad we can't just permanently sacrifice China.
- dnsworks 16y ago
  There's an anti-DDOS service out there, Progent or something. Part of their solution adds all of APNIC's ip addresses into your router's bogon list when you're under attack. It turns out for most DDOS's if you turn off china, the attack just ends.
  - tptacek 16y ago
    I think you're thinking of Prolexic, and my recommendation is that you take any of their claims with a heaping helping of salt. Things may have changed in the 5 years since I've totally forgotten everything I knew about DoS mitigation, but it wasn't the case then that you could simply black hole China to evade them. A more common mechanism used by real networks to mitigate these attacks in production is to pinpoint the target of the attack and offramp its traffic in the ISP core to a "regional scrubbing center" where advanced filtering tools (for instance, packet filters that can handle ACLs with high tens of thousands of terms) can try to sort through the crap.
    - jacquesm 16y ago
      At my colo that is exactly the sort of strategy they use, they have bought a bunch of routers that use FPGA based filters for that. The one scenario I can think of where that might be a problem is if they'd start flooding all the known hosts in a network for the specific purpose of overwhelming the routers. And even those hardware based filtering tools have upper limits.
      - dnsworks 16y ago
        There's really no need to do that. A large enough BotNet can take out almost any host doing nothing more than connecting and doing a GET / HTTP/1.1. Few websites can handle a sudden surge in traffic from 100,000 bots.
        jacquesm 16y ago
        The idea behind an ACL for such an attack is that the same hosts will be used over and over again, taken from a large pool of zombies. So, let's take your example of 100,000 bots, an ACL in the upstream router (as seen from the host) could be used to checked against to identify those packets from the zombies and discard them. After all, a single attack of all 100,000 bots at once will just bring the host to its knees for the time-out of the connections and then it will bounce back up again. So, to increase the effectiveness of the attack they reconnect after every lost connection asking for another resource. If they're smart they'll vary agent strings and other characteristics to make it hard to narrow down who is and who is not legit. Initially you don't have much to go on during such an attack and packet filtering is a reasonably expensive operation when you want to do it for a large number of hosts. So the strategy is to route all the traffic destined for that particular host through a router that has ACLs that are large enough to hold the total IP list for the botnet that is attacking the host, as these IPs become identified. You don't want to route all your traffic through there because then you'd have to do the relatively expensive filtering on all of the packets, even those not destined for that particular host. Now if an attacker were targeting the hosting facility they could thwart this strategy by sending requests to a larger number of hosts in the network in order to make life much harder for the crew fighting the attack. After all, you can't partition the problem anymore in to a portion that is targeted to the host and 'normal' traffic, effectively all the traffic could be bot traffic or it could be normal traffic, for all the receiving hosts. To be able to partition the problem into a smaller one where you can let say 90% or more of the traffic through unfiltered and only concentrate on the remaining 10% would make solving it a bit easier. On the other hand if the attackers are silly enough to re-use the same bots to attack different hosts they've actually given you a clue as to which IPs are bots. I hope that makes sense :)
- vkdelta 16y ago
  If we want, we can :)
- mrtron 16y ago
  It is easy to say that now - would you have said that about US cable modem accounts 10 years ago?
tptacek 16y ago
From 2001-2003 I led development on Arbor Peakflow DoS, which (when I left, at least) was far and away the most popular tool tier-1 ISPs used to manage these kinds of attacks, and my the experience I can relate is: * These attacks happen with shocking frequency (the major incentive ISPs have to mitigate them is not angry customers, but rather the drag it creates on their third-tier engineering staff to hunt down DDoS sources and craft ACLs for them) * There is actually not a whole hell of a lot you can do about them, even if you light up a whole tier 1 core network with mitigation tools. At the end of the day, a well-crafted DDoS attack looks pretty much identical to normal traffic; if you can black-hole China to defeat an attack, it wasn't very cluefully done.
- makmanalp 16y ago
  Regarding your statement that it wasn't very cluefully done, isn't that sort of a hit and miss area? You can't really control much where your malware spreads (and thus where your bots are). As a sidenote, I've been reading your posts for a while now and it seems you always have something insightful to say - cheers!
  - peterwwillis 16y ago
    it's not difficult to control where malware spreads. directly attacking specific ranges is one easy method, as is coding the limitation into your bot or only sending commands to the ones in a certain range.

leftnode 16y ago

No wonder most of my sites were down or not functioning very well this morning. Hope they get everything operational soon, DNSMadeEasy is a great provider.

deleted 16y ago

[deleted]

cliffchang 16y ago

Can anyone give an idea of how large, compared to other DDOS attacks, 50 GBPS is?

dnsworks 16y ago
50gbps is pretty massive. Clearly somebody did something horrifically insulting to China like mention that they could maybe possible consider treating Tibet a little bit more humanely and stop turning it into one big whorehouse.. make a statement like that, and all of china ddos's you.
- dogas 16y ago
  Sarcasm aside, is that really what motivates people in china to regularly pull off attacks like this?
  - paulitex 16y ago
    No. (having lived in China for multiple years) Pretty much all the political issues the West thinks of WRT China are not even given a cursory thought by most Chinese. China just isn't very political. Taiwan is basically a non-issue. Tibet is completely a non-issue. More to the topic: It's much more likely that the motivations are financial.
    - dnsworks 16y ago
      And who do you think pays these people to ddos sites like slideshare and posterous? (both of whom have been my customers during massive ddos's from China). I'm sure there's like one posting somewhere on Posterous that someone in the Chinese government didn't like, so they paid their team of script kiddies for time on their botnets to bully Posterous around. The same thing happened to Slideshare a few years ago. It's about China bullying people around and trying to censor the internet.
      - jhancock 16y ago

wehriam 16y ago

What motivates an attack like this?

Steve0 16y ago
Probably some motive against one of the customers. If you can extort some cash out of a gambling site with short dns-ttl it could be worth your effort. Maybe the target's data center is more ddos-proof than the one from easydns, extortionists go for the weakest link.
- mrtron 16y ago
  Back in the olden days of IRC, there was times when entire regions of the internet were taken offline resulting from a ddos for personal vendettas. For exactly that reason too - they attack the weakest link and often it isn't the host directly. In this case - it really could be anything. The cost of one of these attacks is next to zero. Rarely will the botnet owner lose any machines resulting from an attack. The unfortunate thing after one of these attacks is you have no way of preventing it or going after the source.

quizbiz 16y ago

Can someone please explain to me what DNSMadeEasy actually does? Is a domain provider like 1and1 a client of theirs or is my hosting provider a client? This entry on Hacker News led me to wikipedia where I went from an article about Level3 to an article about Tier 1 Network to Internet Backbone. I feel like I'm on the verge of understanding more about how the internet works but it's all a bit above my head.

geocar 16y ago
They're an anycast dns provider- basically the only kind of third-party DNS service that's worth anything at all. They have some neat domains-related tools that hook into their infrastructure that's kindof a pain to do yourself (even if you've got your own anycast-capable network)

FooBarWidget 16y ago

I'm wondering what the implications of DDOS are for website owners. What if you're on EC2 for example, will you be charged for the 300 TB of traffic? If so that would be an easy way to bankrupt a startup.

dennisgorelik 16y ago
The implications for website owners are that their web sites were temporary unavailable. Traffic simply does not reach the web site. Even normal traffic. For example, you type in your browser www.mywebsite.com. DNSMadeEasy would normally resolve it to your IP address (e.g. 111.222.123.12). But because of the DDoS attach -- mywebsite.com cannot be resolved into any IP address and you cannot open www.mywebsite.com at all.
- dan_manges 16y ago
  That's the implication is there is a DDoS attack on your DNS provider, but I think FooBarWidget was inquiring about the implications of a DDoS attack on your website, and specifically if you would be charged for the bandwidth consumed by the attack.
  - jacquesm 16y ago
    You can - but not everywhere - negotiate that the service you pay for includes protection against DDOS attacks and that it's up to your provider to protect you. You'll pay a larger fee per mbit because they'll need to do more work for you in case you get hit but it might be worth it.
peterwwillis 16y ago
get akamized. you can also build traffic limits into your web stack or OS or network gear if you're afraid of traffic ramps hurting your wallet. like tptacek said, a good DDOS looks like normal traffic, so this could happen if you got slashdotted by 10 different news sites.
nhnifong 16y ago
My policy for this is to simply lock the doors and hide for an hour if my network traffic averages over 2 Mbps for 5 minutes. I would also send myself an email. So far this has never happened.

ximeng 16y ago

(Brief) discussion on webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=970837&highlight=china http://www.webhostingtalk.com/showthread.php?t=970837&hi... Another 30 gbps attack, discussion on webhostingtalk.com: http://www.webhostingtalk.com/showthread.php?t=966658&highlight=china http://www.webhostingtalk.com/showthread.php?t=966658&hi... Edit: interesting thread, apparently the reason it's 30gbps is that it's maxing out the link from China telecom, so legitimate Chinese customers are getting traffic dropped. FBI are involved. Suggested solution by some is to get traffic routed over more intercontinental links possibly via peering agreement with China Telecom, and beyond that political pressure.

petercooper 16y ago

I wonder if, long term, DDOS attacks will be to net neutrality as spam was to nice-and-easy e-mail setups. Now ISPs are running spam blacklists, blocking entire other ISPs at times, and it's a nightmare to run your own SMTP server and deliver mail reliably without jumping through hoops. If DDOS attacks become more and more annoying, they could be used as an excuse to violate net neutrality.

SoftwareMaven 16y ago
I'm not clear on how that would work. A good DDoS attack is nearly indistinguishable from normal traffic. How would net neutrality play into it?
- petercooper 16y ago
  Really good spam e-mail is nearly indistinguishable from normal e-mail. Crazy layers of whitelisting/blacklisting/new DNS settings/laws/software and policies have been piled on top of the mail system to reduce the nefarious effects of spam sent en masse. While spam and DDOS attacks aren't directly comparable in terms of what they are, I'm speculating in terms of what negative effects continued and escalating DDOS attacks could have again in terms of laws, policies, white/blacklisting of entire networks/countries, and so forth.

est 16y ago

The GFW of China has a known vulnerability to amplify a UDP traffic to 2x to 3x as much.

meric 16y ago

I wonder if they're going to change the text on their website: "A DNS service with a 99.9999% uptime history is just the start! DNS Made Easy is so confident of it's uptime record that we offer the best service level agreement in the industry. That is why all businesses that require stable DNS decide to use DNS Made Easy. We have an industry leading 100% uptime gaurantee and will credit all accounts 500% of the downtime." Maybe its a rival DNS provider behind the attack?