3 ms·

>cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet.

>cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day. DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve. This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.

greatquux 12y ago
True, I was thinking of this earlier. When making changes you have to do it in 2 places, which is really the only downside, but it does protect you a bit. Of course, it's still not foolproof - when the first or second DNS server gets used is implementation dependent. We got bit by this, but not too badly - RAX filtered out some residential ISPs whose IPs were part of the DDoS attack but also sent legitimate queries so it was only some people who couldn't resolve. So I might still be too lazy to implement it for all our clients' zones. I'll have to read their post-mortem and see what steps they are taking to avoid this issue in the future.
zzzcpan 12y ago
Hm. I always thought DDoS protection for DNS was rather simple problem to solve, since anycast works with DNS traffic pretty much by design, so you can have hundreds of machines spread all over the world, handling hundreds of gigabits of bandwidth. And filtering of DNS packets is also relatively simple thing to do at line rate.
- skuhn 12y ago
  It's simple except that you need to have capacity greater than what is achievable by people controlling the DDOS. I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity. Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router. So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide. It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
  - phil21 12y ago
    Good post. I would argue some of your locations and how many servers are actually needed, but that's splitting hairs. The real point I want to make is that the only companies really putting this amount of money into DNS infrastructure are DNS providers themselves. I know of no hosting provider/ISP/etc. who puts nearly this much thought or effort into DNS. Even for myself - and I consider myself a DNS junkie - my anycast cluster was a dozen or so servers worldwide co-located w/ a mid-market CDN provider (plus a few locations from my own ASN). That setup was greater than 99% of DNS installs out there - but it would have fallen over under a major attack. It truly takes some skill and financial commitment to build out correctly.
    - zzzcpan 12y ago
      I think OVH does that for DNS, but they already have infrastructure in place.
    - skuhn 12y ago
      I agree with you, there are only two kinds of companies with the kind of DNS infrastructure that could withstand a large scale DDOS: 1. DNS hosting companies (Dyn, Amazon, etc.) 2. Large Internet companies (Yahoo, Google, etc.) And for sure not all of them! A former employer's public zone was hosted by a DNS hosting company that didn't use Anycast, didn't have a footprint outside US/EU, and seemed to have little to no idea when or why traffic spikes happened. So to do DNS really right you need a fair amount of money, some specialized expertise and knowledge of some non-DNSy things (Anycast, among others). I'm really not interested in hosting my own (public) DNS these days at work, it's just not worth the time and effort until you get gigantic. As with e-mail, there are specialists who can do it better than (most) places can. I do kind of worry about depending on one DNS provider, so I've tried to use two when possible. That's a whole other world of hurt though.
- phil21 12y ago
  It's not simply having enough aggregate capacity though either. The way anycast works is that a host will take the shortest network path to the closest server. So for example (very high-level) you might have a server in the US, and one in Europe. If your are eating a DDoS attack maybe the botnet has tons of US-based IPs, but very few euro. In that scenario, you will have all your US customers down but the EU guys are just fine. That's pretty simplistic though. It's less by country, more by ASN. And it's also a case of automation killing you. What happens when the DDoS attack takes out a single PoP worth of DNS servers? Usually you have some automated tooling that will take down that PoP entirely, and withdraw the announcement. You want this behaviour in almost all cases. However, now that attack traffic just got sent to the next-closest server (in this example EU - taking down the whole worldwide cluster). So it's not as simple as having "100gbps worth of inbound DNS capacity" - you need to spread it out properly, and of course deal with attacks like you would otherwise. The other problem is it's UDP - so spoofed packets and the like are a much bigger problem.
  - skuhn 12y ago
    You're right, you definitely need significant single site capacity (2-3 sites with 100Gbit/s, not just the entire system, for instance), and you may even have to play routing games to move traffic around if that's not enough. Any other sites may have to get shut off during massive abuse incidents. It's a tough call to say if it's better to protect the rest of the network by trying to isolate the attack, even though that means you are totally down in one region. I'm inclined to keep 1/2 running than to have nothing, but if I have this release valve of moving traffic to the farms designed to soak up abuse traffic then it makes my decision easier. I should be able to get back up worldwide, if I've done my provisioning properly. One of the nice things about this kind of attack is that your attacker has no choice but to follow your routing, whereas if you use your DNS GSLB to re-route an attacker away from your HTTP server farm (to handle a different sort of attack), they may not stop hammering on its IP address. After all, it's pretty unlikely that they respect DNS TTLs. This approach is simplistic, but it's basically the only solution. The nature of a DDOS attack is that you cannot predict its origin, beyond the distribution of compromised client machines around the world. E.g. your attackers are pretty unlikely to come en masse from Africa. Spoofing is an issue. Not much to do about that at your farm level, but you can start working with your upstreams to get ACLs in place to prevent some portion of that. And if the attack is prolonged and sufficiently large, the only way you're going to stop it is to identify sources and get help from involved networks and ISPs.

greatquux 12y ago

True, I was thinking of this earlier. When making changes you have to do it in 2 places, which is really the only downside, but it does protect you a bit. Of course, it's still not foolproof - when the first or second DNS server gets used is implementation dependent. We got bit by this, but not too badly - RAX filtered out some residential ISPs whose IPs were part of the DDoS attack but also sent legitimate queries so it was only some people who couldn't resolve. So I might still be too lazy to implement it for all our clients' zones. I'll have to read their post-mortem and see what steps they are taking to avoid this issue in the future.

zzzcpan 12y ago

Hm. I always thought DDoS protection for DNS was rather simple problem to solve, since anycast works with DNS traffic pretty much by design, so you can have hundreds of machines spread all over the world, handling hundreds of gigabits of bandwidth. And filtering of DNS packets is also relatively simple thing to do at line rate.

skuhn 12y ago
It's simple except that you need to have capacity greater than what is achievable by people controlling the DDOS. I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity. Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router. So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide. It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
- phil21 12y ago
  Good post. I would argue some of your locations and how many servers are actually needed, but that's splitting hairs. The real point I want to make is that the only companies really putting this amount of money into DNS infrastructure are DNS providers themselves. I know of no hosting provider/ISP/etc. who puts nearly this much thought or effort into DNS. Even for myself - and I consider myself a DNS junkie - my anycast cluster was a dozen or so servers worldwide co-located w/ a mid-market CDN provider (plus a few locations from my own ASN). That setup was greater than 99% of DNS installs out there - but it would have fallen over under a major attack. It truly takes some skill and financial commitment to build out correctly.
  - zzzcpan 12y ago
    I think OVH does that for DNS, but they already have infrastructure in place.
  - skuhn 12y ago
    I agree with you, there are only two kinds of companies with the kind of DNS infrastructure that could withstand a large scale DDOS: 1. DNS hosting companies (Dyn, Amazon, etc.) 2. Large Internet companies (Yahoo, Google, etc.) And for sure not all of them! A former employer's public zone was hosted by a DNS hosting company that didn't use Anycast, didn't have a footprint outside US/EU, and seemed to have little to no idea when or why traffic spikes happened. So to do DNS really right you need a fair amount of money, some specialized expertise and knowledge of some non-DNSy things (Anycast, among others). I'm really not interested in hosting my own (public) DNS these days at work, it's just not worth the time and effort until you get gigantic. As with e-mail, there are specialists who can do it better than (most) places can. I do kind of worry about depending on one DNS provider, so I've tried to use two when possible. That's a whole other world of hurt though.
phil21 12y ago
It's not simply having enough aggregate capacity though either. The way anycast works is that a host will take the shortest network path to the closest server. So for example (very high-level) you might have a server in the US, and one in Europe. If your are eating a DDoS attack maybe the botnet has tons of US-based IPs, but very few euro. In that scenario, you will have all your US customers down but the EU guys are just fine. That's pretty simplistic though. It's less by country, more by ASN. And it's also a case of automation killing you. What happens when the DDoS attack takes out a single PoP worth of DNS servers? Usually you have some automated tooling that will take down that PoP entirely, and withdraw the announcement. You want this behaviour in almost all cases. However, now that attack traffic just got sent to the next-closest server (in this example EU - taking down the whole worldwide cluster). So it's not as simple as having "100gbps worth of inbound DNS capacity" - you need to spread it out properly, and of course deal with attacks like you would otherwise. The other problem is it's UDP - so spoofed packets and the like are a much bigger problem.
- skuhn 12y ago
  You're right, you definitely need significant single site capacity (2-3 sites with 100Gbit/s, not just the entire system, for instance), and you may even have to play routing games to move traffic around if that's not enough. Any other sites may have to get shut off during massive abuse incidents. It's a tough call to say if it's better to protect the rest of the network by trying to isolate the attack, even though that means you are totally down in one region. I'm inclined to keep 1/2 running than to have nothing, but if I have this release valve of moving traffic to the farms designed to soak up abuse traffic then it makes my decision easier. I should be able to get back up worldwide, if I've done my provisioning properly. One of the nice things about this kind of attack is that your attacker has no choice but to follow your routing, whereas if you use your DNS GSLB to re-route an attacker away from your HTTP server farm (to handle a different sort of attack), they may not stop hammering on its IP address. After all, it's pretty unlikely that they respect DNS TTLs. This approach is simplistic, but it's basically the only solution. The nature of a DDOS attack is that you cannot predict its origin, beyond the distribution of compromised client machines around the world. E.g. your attackers are pretty unlikely to come en masse from Africa. Spoofing is an issue. Not much to do about that at your farm level, but you can start working with your upstreams to get ACLs in place to prevent some portion of that. And if the attack is prolonged and sufficiently large, the only way you're going to stop it is to identify sources and get help from involved networks and ISPs.