6 ms·

DDoS Attack Against Default DNS System v2

64 points by hboon 12y ago

tedchs 12y ago
If folks are looking for an alternative DNS host, there is Google Cloud DNS, which I have been using for a while via Google Domains and it's pretty awesome. Conceptually similar to Route53. Can use Google Cloud DNS without using other Google Cloud Platform services. https://cloud.google.com/dns/docs https://cloud.google.com/dns/docs
- mrsaint 12y ago
  >If folks are looking for an alternative DNS host, there is Google Cloud DNS, which I have been using for a while via Google Domains and it's pretty awesome. It's pretty easy to setup too. [1] For my purposes cheaper than Route53. [1] https://www.zeitgeist.se/2014/05/01/google-cloud-dns-how-to/ https://www.zeitgeist.se/2014/05/01/google-cloud-dns-how-to/
  - donavanm 12y ago
    I'm curious what you're doing that's substantially cheaper with Google Cloud DNS. The base pricing looks the same, except Google being $0.2/month and AWS $0.25/month per hosted zone. Also keep in mind that queries for "intra AWS"/ALIAS record queries are free on Route 53.
tedchs 12y ago
It seems like DDoS attacks are becoming more prevalent. Are there forums where attacked companies can share data about the attacks they see and what they had to do to mitigate them? I feel like there should be public community resources around DDoS attacks like there are around other security vulnerabilities.
- AznHisoka 12y ago
  Woudln't this also benefit the attackers now that they know how ppl are going to mitigate them?
  - tedchs 12y ago
    I'm pretty sure we can assume the attackers know how to mitigate these. Just like home burglars know how to prevent burglaries.
- Aeoxic 12y ago
  I manage a private combination of communications bulletins and attack monitoring for a couple of smaller communities / organisations that get attacked on the regular (mostly hosting providers and gaming organisations). I've been considering letting it go invitation-only public.
  - tedchs 12y ago
    Wow, I think it would be awesome to publish some analysis of lessons learned + recovery playbook.
    - Aeoxic 12y ago
      I'll look into cleaning everything up a bit and formalising reporting and perhaps release it properly. A lessons-learned whitepaper would be radical, I'll chat to a few of the people that I run this with and see what they think.
IgorPartola 12y ago
Last time this happened, we had a pretty bad time. NS and SOA TTL being set at 24 hours made switching to Route53 rather difficult. Hopefully, they'll recover soon. Anyone know the details of how these things happen?
- colmmacc 12y ago
  FYI: You can invalidate Public DNS's and OpenDNS's caches manually: https://developers.google.com/speed/public-dns/cache https://developers.google.com/speed/public-dns/cache http://cachecheck.opendns.com/ http://cachecheck.opendns.com/
teach 12y ago
Wondered why my site was spotty. I love Namecheap as a registrar, but I'm questioning now whether I should be using them as my DNS provider. Any suggestions?
- hboon 12y ago
  I'm using DNS Made Easy for a few of my domains. There's a couple of good alternatives.
  - troels 12y ago
    Second that. I don't use them currently, but have in the past. dnsmadeeasy are solid.
- breakingcups 12y ago
  I can fully recommend Point (https://pointhq.com/ https://pointhq.com/), haven't had a single problem yet and support thus far has been very good. The web interface is easy to use and they offer both an API as well as the ability to export zone files. My only gripe would be that the claimed support response times as well as the 'call-me-back' button don't work as advertised.
  - gaadd33 12y ago
    There doesn't seem to be any sort of information about their network, is it multicast? Geo distributed across the world? They mention "Access distributed nameservers across the UK and United States.", does that mean they have 1 VPS in each country serving DNS and that's it? Seems like they have a nice UI but the lack of technical details would make me a bit wary of it.
    - breakingcups 12y ago
      I don't know why they still have that information there... The nameservers I use of them are in: Mountain View, US Dallas, US Amsterdam, The Netherlands Maidenhead, UK Asia Singapore Frankfurt, Germany Paris, France And I know they have at least 5 more.
benmorris 12y ago
I'm going to move some things over to Route 53. I like Namecheap but this is the second DNS issue they've had in a month that has affected several of my sites.
- nullrouted 12y ago
  Why not use cloudflare? It is free and has ipv6 baked in.
  - benmorris 12y ago
    Good point, I will have to check that out
  - cmstoken 12y ago
    Anything to keep in mind when switching to cloudflare? Is it completely free?
    - nullrouted 12y ago
      Nope, it works well. You can just use their DNS service.
      - Someone1234 12y ago
        People have claimed (on HK) that their free tier slows down your site, any truth to those claims in your experience?
        brandonwamboldt 12y ago
        As somebody with sites using all three tiers of their service, I haven't noticed a speed difference between any of the tiers.
        computer 12y ago
        I've had that experience (with Cloudflare in general, not DNS). My average response times as measured by Google webmaster tools went from 30ms to 300ms, if I remember correctly. This way ~18 months ago though, so I don't know if it has changed. And it's to be expected to be slightly slower, at least for dynamic requests, since it's a reverse proxy. I doubt it's limited to the free tier. I've used the DNS for a few years without any issues whatsoever, so I'd recommend them for that.
matthewrussell 12y ago
This is a pretty big and complex attack. We have a ton of DDoS mitigation in place but it has taken some time for the filtering to become effective. We are mitigating this as fast as we possibly can. I and we share your pain and frustration at the inconvenience this causes.
- nullrouted 12y ago
  For those who don't know Matthew Russell is part of NameCheap.
  - matthewrussell 12y ago
    I am, yes. I'll check HN as we restore service and update will now go out on our status page every 20 minutes. My apologies once again for the inconvenience this causes.
matthewrussell 12y ago
We have mitigated over 75% of the attack at this time. Most servers are performing well. I appreciate my responses here are a little obscure but we do practice security through obscurity so I am not going to get into specifics that can be used against us in any future attack. I will say we employ a range of technologies, internal and external, a ddos defense/mitigation. This does include CloudFlare
myers 12y ago
This would be made easier in Namecheap offered Secondary nameserver support. For example, DNS host at Namecheap, but use Linnode as a secondary.
kogir 12y ago
In the age of DNS providers with APIs, you really should have at least two. At YC we've just moved everything to Route53 and will be adding more - likely CloudFlare and Google Cloud DNS. All of it will be managed by configuration files in version control, and won't require anything antiquated like zone transfers. If the code is clean enough I'll release it with pluggable registrar and DNS provider modules.
- randall 12y ago
  Please. The zone transfer thing is what's been the most burdensome. We'd love to have a github repo that we could create a post commit hook to tell a bot to update the syncing.
  - colmmacc 12y ago
    https://dns-api.com/ https://dns-api.com/ is a Git based Route 53 reseller (https://dns-api.com/docs/ https://dns-api.com/docs/). Netflix's denominator is also a great project for managing multi-provider pushes: https://github.com/Netflix/denominator https://github.com/Netflix/denominator . It even supports advanced record types, such as geo and weighted sets.

64 points by hboon 12y ago

If folks are looking for an alternative DNS host, there is Google Cloud DNS, which I have been using for a while via Google Domains and it's pretty awesome. Conceptually similar to Route53. Can use Google Cloud DNS without using other Google Cloud Platform services. https://cloud.google.com/dns/docs https://cloud.google.com/dns/docs

mrsaint 12y ago
>If folks are looking for an alternative DNS host, there is Google Cloud DNS, which I have been using for a while via Google Domains and it's pretty awesome. It's pretty easy to setup too. [1] For my purposes cheaper than Route53. [1] https://www.zeitgeist.se/2014/05/01/google-cloud-dns-how-to/ https://www.zeitgeist.se/2014/05/01/google-cloud-dns-how-to/
- donavanm 12y ago
  I'm curious what you're doing that's substantially cheaper with Google Cloud DNS. The base pricing looks the same, except Google being $0.2/month and AWS $0.25/month per hosted zone. Also keep in mind that queries for "intra AWS"/ALIAS record queries are free on Route 53.

tedchs 12y ago

It seems like DDoS attacks are becoming more prevalent. Are there forums where attacked companies can share data about the attacks they see and what they had to do to mitigate them? I feel like there should be public community resources around DDoS attacks like there are around other security vulnerabilities.

AznHisoka 12y ago
Woudln't this also benefit the attackers now that they know how ppl are going to mitigate them?
- tedchs 12y ago
  I'm pretty sure we can assume the attackers know how to mitigate these. Just like home burglars know how to prevent burglaries.
Aeoxic 12y ago
I manage a private combination of communications bulletins and attack monitoring for a couple of smaller communities / organisations that get attacked on the regular (mostly hosting providers and gaming organisations). I've been considering letting it go invitation-only public.
- tedchs 12y ago
  Wow, I think it would be awesome to publish some analysis of lessons learned + recovery playbook.
  - Aeoxic 12y ago
    I'll look into cleaning everything up a bit and formalising reporting and perhaps release it properly. A lessons-learned whitepaper would be radical, I'll chat to a few of the people that I run this with and see what they think.

IgorPartola 12y ago

Last time this happened, we had a pretty bad time. NS and SOA TTL being set at 24 hours made switching to Route53 rather difficult. Hopefully, they'll recover soon. Anyone know the details of how these things happen?

colmmacc 12y ago
FYI: You can invalidate Public DNS's and OpenDNS's caches manually: https://developers.google.com/speed/public-dns/cache https://developers.google.com/speed/public-dns/cache http://cachecheck.opendns.com/ http://cachecheck.opendns.com/

teach 12y ago

Wondered why my site was spotty. I love Namecheap as a registrar, but I'm questioning now whether I should be using them as my DNS provider. Any suggestions?

hboon 12y ago
I'm using DNS Made Easy for a few of my domains. There's a couple of good alternatives.
- troels 12y ago
  Second that. I don't use them currently, but have in the past. dnsmadeeasy are solid.
breakingcups 12y ago
I can fully recommend Point (https://pointhq.com/ https://pointhq.com/), haven't had a single problem yet and support thus far has been very good. The web interface is easy to use and they offer both an API as well as the ability to export zone files. My only gripe would be that the claimed support response times as well as the 'call-me-back' button don't work as advertised.
- gaadd33 12y ago
  There doesn't seem to be any sort of information about their network, is it multicast? Geo distributed across the world? They mention "Access distributed nameservers across the UK and United States.", does that mean they have 1 VPS in each country serving DNS and that's it? Seems like they have a nice UI but the lack of technical details would make me a bit wary of it.
  - breakingcups 12y ago
    I don't know why they still have that information there... The nameservers I use of them are in: Mountain View, US Dallas, US Amsterdam, The Netherlands Maidenhead, UK Asia Singapore Frankfurt, Germany Paris, France And I know they have at least 5 more.

benmorris 12y ago

I'm going to move some things over to Route 53. I like Namecheap but this is the second DNS issue they've had in a month that has affected several of my sites.

nullrouted 12y ago
Why not use cloudflare? It is free and has ipv6 baked in.
- benmorris 12y ago
  Good point, I will have to check that out
- cmstoken 12y ago
  Anything to keep in mind when switching to cloudflare? Is it completely free?
  - nullrouted 12y ago
    Nope, it works well. You can just use their DNS service.
    - Someone1234 12y ago
      People have claimed (on HK) that their free tier slows down your site, any truth to those claims in your experience?
      - brandonwamboldt 12y ago
        As somebody with sites using all three tiers of their service, I haven't noticed a speed difference between any of the tiers.
      - computer 12y ago
        I've had that experience (with Cloudflare in general, not DNS). My average response times as measured by Google webmaster tools went from 30ms to 300ms, if I remember correctly. This way ~18 months ago though, so I don't know if it has changed. And it's to be expected to be slightly slower, at least for dynamic requests, since it's a reverse proxy. I doubt it's limited to the free tier. I've used the DNS for a few years without any issues whatsoever, so I'd recommend them for that.

matthewrussell 12y ago

This is a pretty big and complex attack. We have a ton of DDoS mitigation in place but it has taken some time for the filtering to become effective. We are mitigating this as fast as we possibly can. I and we share your pain and frustration at the inconvenience this causes.

nullrouted 12y ago
For those who don't know Matthew Russell is part of NameCheap.
- matthewrussell 12y ago
  I am, yes. I'll check HN as we restore service and update will now go out on our status page every 20 minutes. My apologies once again for the inconvenience this causes.

matthewrussell 12y ago

We have mitigated over 75% of the attack at this time. Most servers are performing well. I appreciate my responses here are a little obscure but we do practice security through obscurity so I am not going to get into specifics that can be used against us in any future attack. I will say we employ a range of technologies, internal and external, a ddos defense/mitigation. This does include CloudFlare

myers 12y ago

This would be made easier in Namecheap offered Secondary nameserver support. For example, DNS host at Namecheap, but use Linnode as a secondary.

kogir 12y ago

In the age of DNS providers with APIs, you really should have at least two. At YC we've just moved everything to Route53 and will be adding more - likely CloudFlare and Google Cloud DNS. All of it will be managed by configuration files in version control, and won't require anything antiquated like zone transfers. If the code is clean enough I'll release it with pluggable registrar and DNS provider modules.

randall 12y ago
Please. The zone transfer thing is what's been the most burdensome. We'd love to have a github repo that we could create a post commit hook to tell a bot to update the syncing.
- colmmacc 12y ago
  https://dns-api.com/ https://dns-api.com/ is a Git based Route 53 reseller (https://dns-api.com/docs/ https://dns-api.com/docs/). Netflix's denominator is also a great project for managing multi-provider pushes: https://github.com/Netflix/denominator https://github.com/Netflix/denominator . It even supports advanced record types, such as geo and weighted sets.