5 ms·

RAX has a 6.3 billion market cap. If an org this size -- specialising in hosting -- cannot field a DDOS-resistant DNS server, who can? I'd like to migrate or e

by Erwin 12y ago

RAX has a 6.3 billion market cap. If an org this size -- specialising in hosting -- cannot field a DDOS-resistant DNS server, who can? I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't seem to even offer zone transfers (the best you can do, I guess, is to use the API to get your records out).

ThinkBeat 12y ago
This is a good question.
msisk6 12y ago
This is surprising. The Rackspace DNS infrastructure is large and robust. It was believed just the bandwidth to the masters alone should be sufficient for even the largest DDOS. So this is either something different or of a vastly larger scale than has been seen before. Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.
- majke 12y ago
  Citation needed.
  - devnull42 12y ago
    Friends on the Backbone DNS Team.
    - msisk6 12y ago
      ;)
drzaiusapelord 12y ago
>cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day. DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve. This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.
- greatquux 12y ago
  True, I was thinking of this earlier. When making changes you have to do it in 2 places, which is really the only downside, but it does protect you a bit. Of course, it's still not foolproof - when the first or second DNS server gets used is implementation dependent. We got bit by this, but not too badly - RAX filtered out some residential ISPs whose IPs were part of the DDoS attack but also sent legitimate queries so it was only some people who couldn't resolve. So I might still be too lazy to implement it for all our clients' zones. I'll have to read their post-mortem and see what steps they are taking to avoid this issue in the future.
- zzzcpan 12y ago
  Hm. I always thought DDoS protection for DNS was rather simple problem to solve, since anycast works with DNS traffic pretty much by design, so you can have hundreds of machines spread all over the world, handling hundreds of gigabits of bandwidth. And filtering of DNS packets is also relatively simple thing to do at line rate.
  - skuhn 12y ago
    It's simple except that you need to have capacity greater than what is achievable by people controlling the DDOS. I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity. Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router. So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide. It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
    - phil21 12y ago
justizin 12y ago
While the discussion you want to have is a valid one, you're opening on a complete logical fallacy. Market cap and technical sophistication, esp at large scale, tend to become divergent. In other words, it is generally assumed that big companies do things poorly. That's the entire notion of startups. I will say this: Dyn can probably do a better job of hosting your DNS than Rackspace, Rackspace is not focused on DNS, and I'm not aware that they even really do anything special except for running lots of big DNS servers. Even when I worked at Rackspace 10y ago, the size of their customer base made them a constant target for DoS attacks. Do you need Dyn? Also probably not. Rackspace has patented DDos prevention technology called RiverGuard, but since being patented only illustrates that it is novel, and not that it is of any quality, that is also meaningless. I will say that, even though I prefer as a past employee to work with other companies, because the things Rackspace is bad at are things they don't give a fuck about solving, they run a damned fine network. Further, as a replacement for / alternative to a colo, it is irrelevant whether a company can run a good DNS infrastructure, because that's not what I'm asking them to do. Rackspace DNS is there for convenience. Cloud providers like AWS use their own cloud infrastructure (we think) to host services like Route53, which are also not infallible, but tend not to _entirely_ go down like this. Rackspace, while being somewhat competetive in the cloud space, doesn't operate anything like Amazon, doesn't do anything in a very distributed manner, and is backing those cloud offerings with what I expect to be vertically scaled DNS servers on top-end Dell hardware behind some kind of load balancers. Money doesn't make you good at things, rather it helps you to not have to worry about the details, so it is inversely true that a company with a substantial market share should be expected not to make the same mistakes as the little guys. They're operating all day, all week, all year under the fallacy that their scale is proof of the quality of their work, and following this logic, they don't have to do any work at all. ;)
- quacker 12y ago
  Are you a disgruntled employee? You sure sound like one. That, plus that you worked there over ten years ago - and that Rackspace didn't even offer a DNS api until a few years ago[1] - makes it very hard to take seriously any of your claims about the nature of their dns infrastructure. All we know is that Rackspace has dns servers in three out of its six datacenters[2], and that they appear not to have always-on DDoS protection in place. Market cap and technical sophistication, esp at large scale, tend to become divergent. You say this, but go on to compare Rackspace unfavorably to Amazon who, at 141.93B (of which AWS is just a part), has over 20x Rackspace's market cap and probably 10x the infrastructure[2][3]. Big companies do things poorly, but then it's hard to have highly-available dns without having lots of infrastructure. Namecheap, Route53, Dyn, DNSimple, and now Rackspace have suffered DDoS attacks with outages of varying severity. Don't act like DDoS prevention is a solved problem and claim technical incompetence. 1: http://www.thewhir.com/web-hosting-news/web-host-rackspace-offers-cloud-dns-beta-to-uk-and-us-customers http://www.thewhir.com/web-hosting-news/web-host-rackspace-o... 2: http://www.rackspace.com/about/datacenters http://www.rackspace.com/about/datacenters 3: http://aws.amazon.com/about-aws/global-infrastructure/ http://aws.amazon.com/about-aws/global-infrastructure/
zrail 12y ago
Amazon. http://aws.amazon.com/route53/ http://aws.amazon.com/route53/

by Erwin 12y ago

ThinkBeat 12y ago

This is a good question.

msisk6 12y ago

This is surprising. The Rackspace DNS infrastructure is large and robust. It was believed just the bandwidth to the masters alone should be sufficient for even the largest DDOS. So this is either something different or of a vastly larger scale than has been seen before. Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.

majke 12y ago
Citation needed.
- devnull42 12y ago
  Friends on the Backbone DNS Team.
  - msisk6 12y ago
    ;)

drzaiusapelord 12y ago

>cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day. DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve. This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.

greatquux 12y ago
True, I was thinking of this earlier. When making changes you have to do it in 2 places, which is really the only downside, but it does protect you a bit. Of course, it's still not foolproof - when the first or second DNS server gets used is implementation dependent. We got bit by this, but not too badly - RAX filtered out some residential ISPs whose IPs were part of the DDoS attack but also sent legitimate queries so it was only some people who couldn't resolve. So I might still be too lazy to implement it for all our clients' zones. I'll have to read their post-mortem and see what steps they are taking to avoid this issue in the future.
zzzcpan 12y ago
Hm. I always thought DDoS protection for DNS was rather simple problem to solve, since anycast works with DNS traffic pretty much by design, so you can have hundreds of machines spread all over the world, handling hundreds of gigabits of bandwidth. And filtering of DNS packets is also relatively simple thing to do at line rate.
- skuhn 12y ago
  It's simple except that you need to have capacity greater than what is achievable by people controlling the DDOS. I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity. Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router. So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide. It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
  - phil21 12y ago

justizin 12y ago

While the discussion you want to have is a valid one, you're opening on a complete logical fallacy. Market cap and technical sophistication, esp at large scale, tend to become divergent. In other words, it is generally assumed that big companies do things poorly. That's the entire notion of startups. I will say this: Dyn can probably do a better job of hosting your DNS than Rackspace, Rackspace is not focused on DNS, and I'm not aware that they even really do anything special except for running lots of big DNS servers. Even when I worked at Rackspace 10y ago, the size of their customer base made them a constant target for DoS attacks. Do you need Dyn? Also probably not. Rackspace has patented DDos prevention technology called RiverGuard, but since being patented only illustrates that it is novel, and not that it is of any quality, that is also meaningless. I will say that, even though I prefer as a past employee to work with other companies, because the things Rackspace is bad at are things they don't give a fuck about solving, they run a damned fine network. Further, as a replacement for / alternative to a colo, it is irrelevant whether a company can run a good DNS infrastructure, because that's not what I'm asking them to do. Rackspace DNS is there for convenience. Cloud providers like AWS use their own cloud infrastructure (we think) to host services like Route53, which are also not infallible, but tend not to _entirely_ go down like this. Rackspace, while being somewhat competetive in the cloud space, doesn't operate anything like Amazon, doesn't do anything in a very distributed manner, and is backing those cloud offerings with what I expect to be vertically scaled DNS servers on top-end Dell hardware behind some kind of load balancers. Money doesn't make you good at things, rather it helps you to not have to worry about the details, so it is inversely true that a company with a substantial market share should be expected not to make the same mistakes as the little guys. They're operating all day, all week, all year under the fallacy that their scale is proof of the quality of their work, and following this logic, they don't have to do any work at all. ;)

quacker 12y ago
Are you a disgruntled employee? You sure sound like one. That, plus that you worked there over ten years ago - and that Rackspace didn't even offer a DNS api until a few years ago[1] - makes it very hard to take seriously any of your claims about the nature of their dns infrastructure. All we know is that Rackspace has dns servers in three out of its six datacenters[2], and that they appear not to have always-on DDoS protection in place. Market cap and technical sophistication, esp at large scale, tend to become divergent. You say this, but go on to compare Rackspace unfavorably to Amazon who, at 141.93B (of which AWS is just a part), has over 20x Rackspace's market cap and probably 10x the infrastructure[2][3]. Big companies do things poorly, but then it's hard to have highly-available dns without having lots of infrastructure. Namecheap, Route53, Dyn, DNSimple, and now Rackspace have suffered DDoS attacks with outages of varying severity. Don't act like DDoS prevention is a solved problem and claim technical incompetence. 1: http://www.thewhir.com/web-hosting-news/web-host-rackspace-offers-cloud-dns-beta-to-uk-and-us-customers http://www.thewhir.com/web-hosting-news/web-host-rackspace-o... 2: http://www.rackspace.com/about/datacenters http://www.rackspace.com/about/datacenters 3: http://aws.amazon.com/about-aws/global-infrastructure/ http://aws.amazon.com/about-aws/global-infrastructure/

zrail 12y ago

Amazon. http://aws.amazon.com/route53/ http://aws.amazon.com/route53/