6 ms·

Rackspace DNS DDOS

79 points by treerunner 12y ago

dubcanada 12y ago
All of my websites using Rackspace DNS where down for around 9 hours. It was not a good start on a Monday. And it was even harder to explain that your websites are up, but not up to a client lol, and there is basically nothing I can do. So yah this marks the point at which I will be using R53 and RS DNS servers.
- colmmacc 12y ago
  Tip: If you're using >=2 providers to survive DDOS attacks, it's best to use no more than two name servers from any one provider. That's because resolvers will generally try up to 3 different name servers before giving up. The "no more than two nameservers" rule means that those attempts will always span at least two providers.
  - dubcanada 12y ago
    Amazon by default gives 4 nameservers, can you only use two?
    - rustyconover 12y ago
      Sure you can.
  - jasoncartwright 12y ago
    I didn't know that. Do you have any more info, or a link to some docs that show this?
- bluedino 12y ago
  Sites on Rackspace servers, DNS through NetSol (not that they are good or bad, but it wasn't RS). Didn't realize anything was going on until seeing this article.
- pbhjpbhj 12y ago
  >but not up to a client lol // Why don't DNS clients fail back to use the most recent good IP address as a default? Indeed how can I set up my Kubuntu desktop to have this behaviour. Most domains probably point to the same IP for years at a time I'd imagine. If that's true it seems very strange that just because example.com's DNS server was offline this one day out of the last 600 that my browser can't "guess" that "123.45.67.89 example.com" is going to work. You'd need to add some security around that to avoid abuse I'm sure but doesn't it seem reasonable? Shouldn't DNS failures only prevent you from getting to a server if the IP has changed??
jlgaddis 12y ago
https://status.rackspace.com/ https://status.rackspace.com/
gtCameron 12y ago
Been dealing with this all morning, finally got all of my zones migrated to Route53, but with the amount of time it takes for DNS changes to propagate we are going to be feeling this all day. Any advice for the future on how to add redundancy to my DNS setup? Is it as simple as maintaining Nameservers on two different providers and pointing to them both on my domain?
- leesalminen 12y ago
  I'd like to know as well.
- benmorris 12y ago
  Some decent suggestions brought up from a few weeks ago here https://news.ycombinator.com/item?id=8716662 https://news.ycombinator.com/item?id=8716662 Seems namecheap and DNsimple have suffered from these attacks lately. I had some sites affected by namecheaps DDOS.
Erwin 12y ago
RAX has a 6.3 billion market cap. If an org this size -- specialising in hosting -- cannot field a DDOS-resistant DNS server, who can? I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't seem to even offer zone transfers (the best you can do, I guess, is to use the API to get your records out).
- ThinkBeat 12y ago
  This is a good question.
- msisk6 12y ago
  This is surprising. The Rackspace DNS infrastructure is large and robust. It was believed just the bandwidth to the masters alone should be sufficient for even the largest DDOS. So this is either something different or of a vastly larger scale than has been seen before. Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.
  - majke 12y ago
    Citation needed.
    - devnull42 12y ago
      Friends on the Backbone DNS Team.
      - msisk6 12y ago
        ;)
- drzaiusapelord 12y ago
  >cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day. DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve. This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
riteshpatel 12y ago
It's nice having cheap/free DNS from people like Rackspace and Amazon, but situations like these make you realise that it's sensible to use a company like Dyn (http://dyn.com/ http://dyn.com/) that are experts in highly-available DNS, rather than something that's a small part of a hosting provider's services. It's easy to forget that you can have redundancy in your load balancers, web servers and databases (replication, multiple data centres, etc), but DNS is how you're found by the rest of the Internet. No DNS resolution = no one reaches your expensive, lovingly-crafted infrastructure.
- blfr 12y ago
  On the other hand, the larger your network, the more capacity you have for dealing with attacks.
- mikegioia 12y ago
  I think Rackspace invests a lot of money into their DNS infrastructure. Yea it's free but its mostly because you're paying so much extra for the servers and support. No DNS doesn't always mean no one reaches your expensive infrastructure. That may be true for websites relying on lots of random traffic, but most if not all of our customers have been to our site before so there's a strong chance that the DNS has already been cached on their computer or router.
  - Khao 12y ago
    But that's no use if you use the default TTL of 2-3 hours that I always see whenever I configure new dns entries. Sensible TTL should be at least 24-48 hours, maybe even more for your returning users to not be affected by those kind of outages.
    - dangrossman 12y ago
      A 24-48 hour TTL means you're hosed for 24-48 hours if your hosting service has an extended outage because you can't point any of those visitors at the IP addreses at another host. With a short TTL on the other hand, the worst-case scenario is that your DNS provider is offline and you have to change nameservers at the registrar. With a hosting outage, you can just point the domains at the new IPs and be back up in minutes to hours.
ericcholis 12y ago
For those looking to export your zone from Rackspace, the rackspace python library (prax) will let you: https://github.com/rackspace/pyrax/blob/master/docs/cloud_dns.md https://github.com/rackspace/pyrax/blob/master/docs/cloud_dn... You could also do it via CURL: https://community.rackspace.com/products/f/25/p/1743/4945#4945 https://community.rackspace.com/products/f/25/p/1743/4945#49...
bluedino 12y ago
It's easy to blame 'large hosting provider' and suggest going with 'specialized provider', but isn't stuff like this really leapfrog between hackers and the good guys? Sure, service X might be able to block a 50 foobit attack but what about when the next vulnerability is found and they can launch 500 foobits of DDoS?
- higherpurpose 12y ago
  > but isn't stuff like this really leapfrog between hackers and the good guys? Said the guy on the Hacker News website.
jimschley 12y ago
We went through this recently at Codeship when our provider, DNSimple, had an outage due to DDoS- https://blog.codeship.com/dnsimple-ddos-outage/ https://blog.codeship.com/dnsimple-ddos-outage/ DNS is a service that often ends up as a single point of failure in infrastructures I've seen as it's non-trivial to implement redundancy. Having a repository/API approach to deploying DNS records saved us in this incident: http://blog.codeship.com/dnsimple-dns-history-continuous-deployment/ http://blog.codeship.com/dnsimple-dns-history-continuous-dep...
- stevekemp 12y ago
  Nice to see you mention a repository - I love storing DNS details in git, and setup a site to push that on to Amazon's route53 infrastructure (https://dns-api.com/ https://dns-api.com/). Having revision-control is wonderful for history-tracking.
  - colinbartlett 12y ago
    This is a really cool idea. I'd like it even more if I could add "remotes" for different providers. DNSimple, Route53, etc. and push my changes to each one. I suppose the service would need to support a number of providers that themselves support APIs.
    - fmotlik 12y ago
      Would also be possible to use the dns_deploy tool one of my team mates wrote (https://github.com/codeship/dns_deploy/commits/master https://github.com/codeship/dns_deploy/commits/master) or by extending Terraform (https://terraform.io/docs/providers/dnsimple/index.html https://terraform.io/docs/providers/dnsimple/index.html)
      - colinbartlett 12y ago
        Hey I wanted to thank you for showing this to me. I am using it on my personal domain and plan to use it on some customers as well. Really cool tool, thanks!
anthony_franco 12y ago
For anyone else worried about this, the best way to mitigate this going forward is to have secondary DNS servers. Your primary DNS provider should allow automatic zone transfers. This makes it so that any changes you do to your primary service gets propagated to the secondary service within seconds. Once setup you'll automatically have redundancy incase the primary provider starts timing out.
alexbecker 12y ago
Who launches these massive DDoS attacks against DNS infrastructure? It would require a substantial botnet to pull off, so they must have some compelling reason. Maybe there's something obvious I'm missing, but I don't see one, except perhaps for a government or large organization which had many competitors using the DNS service they attack.
- zzzcpan 12y ago
  There is no requirement in a substantial botnet for this, even a small one will do. Also internet is full of cheap amplified ddos offerings. Attacker could use multiple offerings for more bandwidth if he feels like it. Maybe this is something obvious you are missing.
- gnopgnip 12y ago
  Until more orgs implement BCP 38, it only takes a handful of servers with an open resolver to multiply the attackers bandwidth by a factor of 30 or more. UDP does not verify the source IP is the sender. Most of the internet will drop spoofed packets already, but there are still enough netblocks passing spoofed traffic, and vulnerable DNS servers for this to be a concern.
codezero 12y ago
I'm not savvy to this stuff so pardon the unsolicited conspiracy theory, does this DDoS have anything to do with the NK internet outage, or is it just coincidence?
- ratsmack 12y ago
  It can be difficult to tell. If the traffic is coming from a botnet, only identifying the Command And Control operators will tell you anything.
philip1209 12y ago
Here is the status page - note that it uses Rackspace DNS, though: https://status.rackspace.com/ https://status.rackspace.com/
king_phil 12y ago
Anyone with an idea how well Google Cloud DNS handles DDoS? I would image it just sucks it up, because they already saw any DDoS volume (would it be any or every, I'm not a native speaker?) before.
some-dude314 12y ago
I'll make the answer simple. Market cap and size do not matter. All it takes is _competent staff_ and management support. Actually DNS is not that hard to mitigate. You just need optimized compute and the bandwidth to take it to the clean up equipment.

79 points by treerunner 12y ago

All of my websites using Rackspace DNS where down for around 9 hours. It was not a good start on a Monday. And it was even harder to explain that your websites are up, but not up to a client lol, and there is basically nothing I can do. So yah this marks the point at which I will be using R53 and RS DNS servers.

colmmacc 12y ago
Tip: If you're using >=2 providers to survive DDOS attacks, it's best to use no more than two name servers from any one provider. That's because resolvers will generally try up to 3 different name servers before giving up. The "no more than two nameservers" rule means that those attempts will always span at least two providers.
- dubcanada 12y ago
  Amazon by default gives 4 nameservers, can you only use two?
  - rustyconover 12y ago
    Sure you can.
- jasoncartwright 12y ago
  I didn't know that. Do you have any more info, or a link to some docs that show this?
bluedino 12y ago
Sites on Rackspace servers, DNS through NetSol (not that they are good or bad, but it wasn't RS). Didn't realize anything was going on until seeing this article.
pbhjpbhj 12y ago
>but not up to a client lol // Why don't DNS clients fail back to use the most recent good IP address as a default? Indeed how can I set up my Kubuntu desktop to have this behaviour. Most domains probably point to the same IP for years at a time I'd imagine. If that's true it seems very strange that just because example.com's DNS server was offline this one day out of the last 600 that my browser can't "guess" that "123.45.67.89 example.com" is going to work. You'd need to add some security around that to avoid abuse I'm sure but doesn't it seem reasonable? Shouldn't DNS failures only prevent you from getting to a server if the IP has changed??

jlgaddis 12y ago

https://status.rackspace.com/ https://status.rackspace.com/

gtCameron 12y ago

Been dealing with this all morning, finally got all of my zones migrated to Route53, but with the amount of time it takes for DNS changes to propagate we are going to be feeling this all day. Any advice for the future on how to add redundancy to my DNS setup? Is it as simple as maintaining Nameservers on two different providers and pointing to them both on my domain?

leesalminen 12y ago
I'd like to know as well.
benmorris 12y ago
Some decent suggestions brought up from a few weeks ago here https://news.ycombinator.com/item?id=8716662 https://news.ycombinator.com/item?id=8716662 Seems namecheap and DNsimple have suffered from these attacks lately. I had some sites affected by namecheaps DDOS.

Erwin 12y ago

RAX has a 6.3 billion market cap. If an org this size -- specialising in hosting -- cannot field a DDOS-resistant DNS server, who can? I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't seem to even offer zone transfers (the best you can do, I guess, is to use the API to get your records out).

ThinkBeat 12y ago
This is a good question.
msisk6 12y ago
This is surprising. The Rackspace DNS infrastructure is large and robust. It was believed just the bandwidth to the masters alone should be sufficient for even the largest DDOS. So this is either something different or of a vastly larger scale than has been seen before. Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.
- majke 12y ago
  Citation needed.
  - devnull42 12y ago
    Friends on the Backbone DNS Team.
    - msisk6 12y ago
      ;)
drzaiusapelord 12y ago
>cannot field a DDOS-resistant DNS server, who can? Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day. DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve. This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]

riteshpatel 12y ago

It's nice having cheap/free DNS from people like Rackspace and Amazon, but situations like these make you realise that it's sensible to use a company like Dyn (http://dyn.com/ http://dyn.com/) that are experts in highly-available DNS, rather than something that's a small part of a hosting provider's services. It's easy to forget that you can have redundancy in your load balancers, web servers and databases (replication, multiple data centres, etc), but DNS is how you're found by the rest of the Internet. No DNS resolution = no one reaches your expensive, lovingly-crafted infrastructure.

blfr 12y ago
On the other hand, the larger your network, the more capacity you have for dealing with attacks.
mikegioia 12y ago
I think Rackspace invests a lot of money into their DNS infrastructure. Yea it's free but its mostly because you're paying so much extra for the servers and support. No DNS doesn't always mean no one reaches your expensive infrastructure. That may be true for websites relying on lots of random traffic, but most if not all of our customers have been to our site before so there's a strong chance that the DNS has already been cached on their computer or router.
- Khao 12y ago
  But that's no use if you use the default TTL of 2-3 hours that I always see whenever I configure new dns entries. Sensible TTL should be at least 24-48 hours, maybe even more for your returning users to not be affected by those kind of outages.
  - dangrossman 12y ago
    A 24-48 hour TTL means you're hosed for 24-48 hours if your hosting service has an extended outage because you can't point any of those visitors at the IP addreses at another host. With a short TTL on the other hand, the worst-case scenario is that your DNS provider is offline and you have to change nameservers at the registrar. With a hosting outage, you can just point the domains at the new IPs and be back up in minutes to hours.

ericcholis 12y ago

For those looking to export your zone from Rackspace, the rackspace python library (prax) will let you: https://github.com/rackspace/pyrax/blob/master/docs/cloud_dns.md https://github.com/rackspace/pyrax/blob/master/docs/cloud_dn... You could also do it via CURL: https://community.rackspace.com/products/f/25/p/1743/4945#4945 https://community.rackspace.com/products/f/25/p/1743/4945#49...

bluedino 12y ago

It's easy to blame 'large hosting provider' and suggest going with 'specialized provider', but isn't stuff like this really leapfrog between hackers and the good guys? Sure, service X might be able to block a 50 foobit attack but what about when the next vulnerability is found and they can launch 500 foobits of DDoS?

higherpurpose 12y ago
> but isn't stuff like this really leapfrog between hackers and the good guys? Said the guy on the Hacker News website.

jimschley 12y ago

We went through this recently at Codeship when our provider, DNSimple, had an outage due to DDoS- https://blog.codeship.com/dnsimple-ddos-outage/ https://blog.codeship.com/dnsimple-ddos-outage/ DNS is a service that often ends up as a single point of failure in infrastructures I've seen as it's non-trivial to implement redundancy. Having a repository/API approach to deploying DNS records saved us in this incident: http://blog.codeship.com/dnsimple-dns-history-continuous-deployment/ http://blog.codeship.com/dnsimple-dns-history-continuous-dep...

stevekemp 12y ago
Nice to see you mention a repository - I love storing DNS details in git, and setup a site to push that on to Amazon's route53 infrastructure (https://dns-api.com/ https://dns-api.com/). Having revision-control is wonderful for history-tracking.
- colinbartlett 12y ago
  This is a really cool idea. I'd like it even more if I could add "remotes" for different providers. DNSimple, Route53, etc. and push my changes to each one. I suppose the service would need to support a number of providers that themselves support APIs.
  - fmotlik 12y ago
    Would also be possible to use the dns_deploy tool one of my team mates wrote (https://github.com/codeship/dns_deploy/commits/master https://github.com/codeship/dns_deploy/commits/master) or by extending Terraform (https://terraform.io/docs/providers/dnsimple/index.html https://terraform.io/docs/providers/dnsimple/index.html)
    - colinbartlett 12y ago
      Hey I wanted to thank you for showing this to me. I am using it on my personal domain and plan to use it on some customers as well. Really cool tool, thanks!

anthony_franco 12y ago

For anyone else worried about this, the best way to mitigate this going forward is to have secondary DNS servers. Your primary DNS provider should allow automatic zone transfers. This makes it so that any changes you do to your primary service gets propagated to the secondary service within seconds. Once setup you'll automatically have redundancy incase the primary provider starts timing out.

alexbecker 12y ago

Who launches these massive DDoS attacks against DNS infrastructure? It would require a substantial botnet to pull off, so they must have some compelling reason. Maybe there's something obvious I'm missing, but I don't see one, except perhaps for a government or large organization which had many competitors using the DNS service they attack.

zzzcpan 12y ago
There is no requirement in a substantial botnet for this, even a small one will do. Also internet is full of cheap amplified ddos offerings. Attacker could use multiple offerings for more bandwidth if he feels like it. Maybe this is something obvious you are missing.
gnopgnip 12y ago
Until more orgs implement BCP 38, it only takes a handful of servers with an open resolver to multiply the attackers bandwidth by a factor of 30 or more. UDP does not verify the source IP is the sender. Most of the internet will drop spoofed packets already, but there are still enough netblocks passing spoofed traffic, and vulnerable DNS servers for this to be a concern.

codezero 12y ago

I'm not savvy to this stuff so pardon the unsolicited conspiracy theory, does this DDoS have anything to do with the NK internet outage, or is it just coincidence?

ratsmack 12y ago
It can be difficult to tell. If the traffic is coming from a botnet, only identifying the Command And Control operators will tell you anything.

philip1209 12y ago

Here is the status page - note that it uses Rackspace DNS, though: https://status.rackspace.com/ https://status.rackspace.com/

king_phil 12y ago

Anyone with an idea how well Google Cloud DNS handles DDoS? I would image it just sucks it up, because they already saw any DDoS volume (would it be any or every, I'm not a native speaker?) before.

some-dude314 12y ago

I'll make the answer simple. Market cap and size do not matter. All it takes is _competent staff_ and management support. Actually DNS is not that hard to mitigate. You just need optimized compute and the bandwidth to take it to the clean up equipment.