5 ms·

UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes

48 points by rdl 11y ago

lorddoig 11y ago
Titular hyperbole, much?
- voltagex_ 11y ago
  Original (current?) title on the page was UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes.
- dang 11y ago
  We changed the title from "UltraDNS takes down the Internet for 90min" to the article's title.
Thaxll 11y ago
How can unicast be DDoS?
- edgan 11y ago
  The article says it was an internal issue, not DDoS or hacking.
edgan 11y ago
I am surprised that Netflix uses UltraDNS when they use AWS and they could be using Route53.
- b1naryth1ef 11y ago
  I would imagine they don't use it for exactly that case. Imagine a case where most of AWS is having critical problems, you would want (at the very least) control of your DNS to attempt a partial recovery of service for some portion of your customers that have providers who respect DNS TTL's.
  - bwblabs 11y ago
    Running my own DNS servers. But having issues with clients using Rackspace Apps - hosted email (still down due to UltraDNS). Would there be anything against using multiple large anycast DNS providers?
  - hueving 11y ago
    Netflix is 100% dependent on AWS for infrastructure. If AWS is having problems, there is nowhere to redirect to.
    - tiagobraw 11y ago
      of course there is, they could redirect to a page explaining the problem instead of a generic error page...
      - deleted 11y ago
        [deleted]
- Thaxll 11y ago
  UltraDNS is more advanced / pro than route53.
  - acdha 11y ago
    Could you provide some details about how and why it's better?
- jread 11y ago
  Amazon.com uses UltraDNS and Dyn
  - semiquaver 11y ago
    Confirmed, this issue more or less took down Amazon's Merchant Web Services API today.
- jedberg 11y ago
  It's because Ultra lets you do geo based routing by state and province and Route53 doesn't (at least not yet)
jonah 11y ago
Interesting. We use UltraDNS and didn't go down at all.
jonah 11y ago
Here are the sequence of emails we as a customer received: 1:44PM PST - The Neustar UltraDNS service is currently experiencing DDoS traffic in the U.S. East Region. The Security Operations Team is currently working on mitigating attack traffic and further updates will be provided as soon as possible. 3:07PM PST - The Neustar UltraDNS service is currently experiencing a major spike in traffic across several regions. The Security Operations Team is conducting routing changes and working to reroute traffic at this time. Customer will experience DNS timeouts and lack of resolution. Further updates will be sent as soon as possible. 4:50PM PST - At 4:18 EDT time, a service outage affected Neustar’s UltraDNS service. The impact was predominately on the East Coast of the U.S., however some impacts could be seen in other U.S regions. Neustar UltraDNS Service was restored at 6:34 EDT time. We are currently investigating the root cause of the outage and will provide an update as more information is available.
- lalt 11y ago
  The data from the outage is all here: https://blog.thousandeyes.com/ultradns-outage-october-2015/ https://blog.thousandeyes.com/ultradns-outage-october-2015/ Check it out.
tshtf 11y ago
Serious question here: Do people really spend $90/month for 2M DNS queries? https://www.neustar.biz/services/dns-services/managed-dns-packages https://www.neustar.biz/services/dns-services/managed-dns-pa...
- bwblabs 11y ago
  And that's even without 'DNSSEC and IPv6 support', 'Regional Routing' and 'DNS Load Balancing (where available)', that's only available for 'UltraDNS Enterprise'... (I'm running my own (Power)DNS, after EveryDNS was bought by DYN and they didn't grandfathered old plans, charging per domain, which made it extremely pricey)
  - mbubb 11y ago
    Yeah - i looked at DYN and they are expensive. The "Internet Intelligence" stuff looks interesting but that is an add on.
    - voltagex_ 11y ago
      I need some dynamic DNS for my VMs, I'm getting away with Avahi/mDNS right now but I need to set up an Active Directory lab. Do you have any recommendations for dynamic DNS? Bonus points if it's BIND9 compatible.
      - jlgaddis 11y ago
        Put AD-joined hosts in a subdomain, e.g. subdomain.example.com. AD DCs are authoritative for the subdomain. Your public facing / authoritative nameservers for example.com should be listed (publicly) as the only NS's for subdomain.example.com and should slave it from your DCs.
- mbubb 11y ago
  More money and more queries-per-month then that at nsone.net. They are very good. The api is especially nice for setting up failover and geo pools.
fletchowns 11y ago
I suppose this was also the cause of the 90 minute Bitbucket outage today?
- dholowiski 11y ago
  Surprisingly, no. Or at least they're not claiming so. They say it was a problem with a database server. http://status.bitbucket.org/ http://status.bitbucket.org/
  - fletchowns 11y ago
    That was a different "Git and Mercurial over HTTPS slow or not responding" outage from yesterday (or "02:00 and 03:00 UTC on 15 October 2015") The more recent one started @ Oct 15, 20:19 UTC, they don't seem to have root cause on the status page for it yet.

48 points by rdl 11y ago

lorddoig 11y ago

Titular hyperbole, much?

voltagex_ 11y ago
Original (current?) title on the page was UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes.
dang 11y ago
We changed the title from "UltraDNS takes down the Internet for 90min" to the article's title.

Thaxll 11y ago

How can unicast be DDoS?

edgan 11y ago
The article says it was an internal issue, not DDoS or hacking.

edgan 11y ago

I am surprised that Netflix uses UltraDNS when they use AWS and they could be using Route53.

b1naryth1ef 11y ago
I would imagine they don't use it for exactly that case. Imagine a case where most of AWS is having critical problems, you would want (at the very least) control of your DNS to attempt a partial recovery of service for some portion of your customers that have providers who respect DNS TTL's.
- bwblabs 11y ago
  Running my own DNS servers. But having issues with clients using Rackspace Apps - hosted email (still down due to UltraDNS). Would there be anything against using multiple large anycast DNS providers?
- hueving 11y ago
  Netflix is 100% dependent on AWS for infrastructure. If AWS is having problems, there is nowhere to redirect to.
  - tiagobraw 11y ago
    of course there is, they could redirect to a page explaining the problem instead of a generic error page...
    - deleted 11y ago
      [deleted]
Thaxll 11y ago
UltraDNS is more advanced / pro than route53.
- acdha 11y ago
  Could you provide some details about how and why it's better?
jread 11y ago
Amazon.com uses UltraDNS and Dyn
- semiquaver 11y ago
  Confirmed, this issue more or less took down Amazon's Merchant Web Services API today.
jedberg 11y ago
It's because Ultra lets you do geo based routing by state and province and Route53 doesn't (at least not yet)

jonah 11y ago

Interesting. We use UltraDNS and didn't go down at all.

jonah 11y ago

Here are the sequence of emails we as a customer received: 1:44PM PST - The Neustar UltraDNS service is currently experiencing DDoS traffic in the U.S. East Region. The Security Operations Team is currently working on mitigating attack traffic and further updates will be provided as soon as possible. 3:07PM PST - The Neustar UltraDNS service is currently experiencing a major spike in traffic across several regions. The Security Operations Team is conducting routing changes and working to reroute traffic at this time. Customer will experience DNS timeouts and lack of resolution. Further updates will be sent as soon as possible. 4:50PM PST - At 4:18 EDT time, a service outage affected Neustar’s UltraDNS service. The impact was predominately on the East Coast of the U.S., however some impacts could be seen in other U.S regions. Neustar UltraDNS Service was restored at 6:34 EDT time. We are currently investigating the root cause of the outage and will provide an update as more information is available.

lalt 11y ago
The data from the outage is all here: https://blog.thousandeyes.com/ultradns-outage-october-2015/ https://blog.thousandeyes.com/ultradns-outage-october-2015/ Check it out.

tshtf 11y ago

Serious question here: Do people really spend $90/month for 2M DNS queries? https://www.neustar.biz/services/dns-services/managed-dns-packages https://www.neustar.biz/services/dns-services/managed-dns-pa...

bwblabs 11y ago
And that's even without 'DNSSEC and IPv6 support', 'Regional Routing' and 'DNS Load Balancing (where available)', that's only available for 'UltraDNS Enterprise'... (I'm running my own (Power)DNS, after EveryDNS was bought by DYN and they didn't grandfathered old plans, charging per domain, which made it extremely pricey)
- mbubb 11y ago
  Yeah - i looked at DYN and they are expensive. The "Internet Intelligence" stuff looks interesting but that is an add on.
  - voltagex_ 11y ago
    I need some dynamic DNS for my VMs, I'm getting away with Avahi/mDNS right now but I need to set up an Active Directory lab. Do you have any recommendations for dynamic DNS? Bonus points if it's BIND9 compatible.
    - jlgaddis 11y ago
      Put AD-joined hosts in a subdomain, e.g. subdomain.example.com. AD DCs are authoritative for the subdomain. Your public facing / authoritative nameservers for example.com should be listed (publicly) as the only NS's for subdomain.example.com and should slave it from your DCs.
mbubb 11y ago
More money and more queries-per-month then that at nsone.net. They are very good. The api is especially nice for setting up failover and geo pools.

fletchowns 11y ago

I suppose this was also the cause of the 90 minute Bitbucket outage today?

dholowiski 11y ago
Surprisingly, no. Or at least they're not claiming so. They say it was a problem with a database server. http://status.bitbucket.org/ http://status.bitbucket.org/
- fletchowns 11y ago
  That was a different "Git and Mercurial over HTTPS slow or not responding" outage from yesterday (or "02:00 and 03:00 UTC on 15 October 2015") The more recent one started @ Oct 15, 20:19 UTC, they don't seem to have root cause on the status page for it yet.