5 ms·

Network Solutions' DNS was down

72 points by mikegirouard 13y ago

simonsarris 13y ago
Honest question from a networking beginner: So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues. If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?
- mgkimsal 13y ago
  yes
- area51org 13y ago
  Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.
  - jeltz 13y ago
    Also you as always have to be careful and make sure that all DNS servers have the same content or else you will get weird bugs.
  - thejosh 13y ago
    Wouldn't a delay be better than full downtime?
  - colmmacc 13y ago
    (I work on Route 53). We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers. In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider. That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.
    - mmmooo 13y ago
      Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.
      - colmmacc 13y ago
        I haven't read the bind source in a long time, but my best guess is that it's in memory. http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.htm http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h... Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.
        mmmooo 13y ago
        interesting, not a ton of details of how. off to the source I go, thanks. sounds like it assigns a 'score' of some sort to each authoritative ns.
        isb 13y ago
        This might help - section 4.4 has formal details on the Bind server selection algorithms: http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theoretical-computer-science-internet-research-problems-spring-2002/lecture-notes/dns.pdf http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theo...
        mmmooo 13y ago
        yep definitely, an interesting implementation, and certainly almost impossible to guess from passive observation. thanks.
        dsl 13y ago
        BIND assigns a score based on every IP it talks to and a hit rate. (so if ns1.foo.com resolves to 3 IPs, each is tracked individually, same if ns1.bar.com and ns1.baz.com are the same IP) It is a pretty small cache, maybe like 1024? This is all from fuzzy 5 year old memory.
  - smountcastle 13y ago
    It depends on the implementation of the resolver, but in general when no information is known about a nameserver, the resolver will pick one at random. Overtime, it'll keep track of the latency and pick one (usually randomly within a band specifying a range of latencies). See this paper for more info: https://www.dns-oarc.net/files/workshop-201203/OARC-workshop-London-2012-NS-selection.pdf https://www.dns-oarc.net/files/workshop-201203/OARC-workshop...
- deleted 13y ago
  [deleted]
- throwit1979 13y ago
  In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.
deleted 13y ago
[deleted]
dholowiski 13y ago
Oh crap. Today is going to be a bad day for me.
- jryce 13y ago
  If the mail server goes down again it'll be a bad day for us too.
  - dholowiski 13y ago
    We run our own mail server, but the MX records come from network solutions :(
    - dholowiski 13y ago
      DNS for our domains seems to be back up now, for now at least. In case anyone is interested, looking at our google analytics it appears this started sometime around 6AM MST, and got 'fixed' around 8:45AM MST. Of course it's impossible to say exactly when, because of DNS TTL & caching.
jryce 13y ago
Their mail server appears to be online.
billsinc 13y ago
Seeing this as well, appears to be DNS only at this point.
- mikegirouard 13y ago
  You're right. Title updated.
esmale 13y ago
I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.
- mikegirouard 13y ago
  Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP. It doesn't matter how long I do this... DNS problems always get me.
  - dedward 13y ago
    Have you thought about setting up with an external DNS monitoring service? It's cheap and really comes in handy over the years.
- astrodust 13y ago
  And to think you paid extra for this service.
supine 13y ago
If your name servers are *.worldnic.com it appears you are affected. First mention on Twitter was an hour ago https://twitter.com/thefrost/status/357483942082920449 https://twitter.com/thefrost/status/357483942082920449
mikegirouard 13y ago
This[0] FB post is the best I can come up with as an explanation. > Yesterday, some Network Solutions customer sites were compromised. The funny thing is, they have a link in their post going back to their site, which of course doesn't work. [0]: https://www.facebook.com/networksolutions/posts/10151466801446020 https://www.facebook.com/networksolutions/posts/101514668014...
- muraiki 13y ago
  That explains my friend's site yesterday. Requests to his site sometimes resulted in the display of a banner page by "Islamic Ghosts Team," but not all the time. I noticed that Network Solutions was apparently running Apache 2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).
theg2 13y ago
We're slowly coming back online, there are reports of a (D)DOS attack against them but I can't find a source.
- TranceMan 13y ago
  Some talk here [1] of a recorded message from their support number that they are currently experiencing a DOS attack. 1. http://www.webhostingtalk.com/showthread.php?t=1285835 http://www.webhostingtalk.com/showthread.php?t=1285835
pilif 13y ago
This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them). When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data. Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.
- esw 13y ago
  This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.
  - pilif 13y ago
    I would assume that sites affected by a Network Solutions downtime would qualify as small(ish) sites because the big ones have other solutions anyways. Also, it's much easier to deal with a DOS on DNS than with a DOS on your app servers. If you can handle it for your app servers, you'll be able to handle if for DNS. If not, well, then you're down anyways.
  - dsl 13y ago
    FYI, Zerigo DNS is a joke. They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707 https://news.ycombinator.com/item?id=5922707
- 13y ago
asr2bd 13y ago
Our site has been sporadically up and down for the past few hours. Glad I know the culprit finally. Hope this gets resolved soon
nodata 13y ago
Site down? Don't link to it. Bad netiquette. Jesus christ.
TranceMan 13y ago
Their facebook page has been updated: Network Solutions is experiencing a Distributed Denial of Service (DDOS) attack that is impacting our customers as well as the Network Solutions site. Our technology team is working to mitigate the situation. Please check back for updates. https://www.facebook.com/networksolutions/posts/10151468105866020 https://www.facebook.com/networksolutions/posts/101514681058...
ottoflux 13y ago
Back up.
SubZero 13y ago
Why did I click on the link to a down website? What did I expect? Shame on me.
Tenhundfeld 13y ago
TechZone360 (ugh, so many ads) article with a little more information: Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/17/345902-network-solutions-experiences-hijacking-dns-records.htm http://www.techzone360.com/topics/techzone/articles/2013/07/...
- muraiki 13y ago
  A comment in the article pointed out that this one seems very similar to a Cisco article from a month ago: http://blogs.cisco.com/security/hijacking-of-dns-records-from-network-solutions/ http://blogs.cisco.com/security/hijacking-of-dns-records-fro...
dangayle 13y ago
Just what I wanted to wake up to. Client emails asking me why their sites were down. Sigh.
dholowiski 13y ago
This appears to be a problem again this afternoon (1:44PM MST)

72 points by mikegirouard 13y ago

Honest question from a networking beginner: So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues. If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?

mgkimsal 13y ago
yes
area51org 13y ago
Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.
- jeltz 13y ago
  Also you as always have to be careful and make sure that all DNS servers have the same content or else you will get weird bugs.
- thejosh 13y ago
  Wouldn't a delay be better than full downtime?
- colmmacc 13y ago
  (I work on Route 53). We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers. In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider. That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.
  - mmmooo 13y ago
    Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.
    - colmmacc 13y ago
      I haven't read the bind source in a long time, but my best guess is that it's in memory. http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.htm http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h... Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.
      - mmmooo 13y ago
        interesting, not a ton of details of how. off to the source I go, thanks. sounds like it assigns a 'score' of some sort to each authoritative ns.
        isb 13y ago
        This might help - section 4.4 has formal details on the Bind server selection algorithms: http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theoretical-computer-science-internet-research-problems-spring-2002/lecture-notes/dns.pdf http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theo...
        mmmooo 13y ago
        yep definitely, an interesting implementation, and certainly almost impossible to guess from passive observation. thanks.
        dsl 13y ago
        BIND assigns a score based on every IP it talks to and a hit rate. (so if ns1.foo.com resolves to 3 IPs, each is tracked individually, same if ns1.bar.com and ns1.baz.com are the same IP) It is a pretty small cache, maybe like 1024? This is all from fuzzy 5 year old memory.
- smountcastle 13y ago
  It depends on the implementation of the resolver, but in general when no information is known about a nameserver, the resolver will pick one at random. Overtime, it'll keep track of the latency and pick one (usually randomly within a band specifying a range of latencies). See this paper for more info: https://www.dns-oarc.net/files/workshop-201203/OARC-workshop-London-2012-NS-selection.pdf https://www.dns-oarc.net/files/workshop-201203/OARC-workshop...
deleted 13y ago
[deleted]
throwit1979 13y ago
In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.

deleted 13y ago

[deleted]

dholowiski 13y ago

Oh crap. Today is going to be a bad day for me.

jryce 13y ago
If the mail server goes down again it'll be a bad day for us too.
- dholowiski 13y ago
  We run our own mail server, but the MX records come from network solutions :(
  - dholowiski 13y ago
    DNS for our domains seems to be back up now, for now at least. In case anyone is interested, looking at our google analytics it appears this started sometime around 6AM MST, and got 'fixed' around 8:45AM MST. Of course it's impossible to say exactly when, because of DNS TTL & caching.

jryce 13y ago

Their mail server appears to be online.

billsinc 13y ago

Seeing this as well, appears to be DNS only at this point.

mikegirouard 13y ago
You're right. Title updated.

esmale 13y ago

I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.

mikegirouard 13y ago
Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP. It doesn't matter how long I do this... DNS problems always get me.
- dedward 13y ago
  Have you thought about setting up with an external DNS monitoring service? It's cheap and really comes in handy over the years.
astrodust 13y ago
And to think you paid extra for this service.

supine 13y ago

If your name servers are *.worldnic.com it appears you are affected. First mention on Twitter was an hour ago https://twitter.com/thefrost/status/357483942082920449 https://twitter.com/thefrost/status/357483942082920449

mikegirouard 13y ago

This[0] FB post is the best I can come up with as an explanation. > Yesterday, some Network Solutions customer sites were compromised. The funny thing is, they have a link in their post going back to their site, which of course doesn't work. [0]: https://www.facebook.com/networksolutions/posts/10151466801446020 https://www.facebook.com/networksolutions/posts/101514668014...

muraiki 13y ago
That explains my friend's site yesterday. Requests to his site sometimes resulted in the display of a banner page by "Islamic Ghosts Team," but not all the time. I noticed that Network Solutions was apparently running Apache 2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).

theg2 13y ago

We're slowly coming back online, there are reports of a (D)DOS attack against them but I can't find a source.

TranceMan 13y ago
Some talk here [1] of a recorded message from their support number that they are currently experiencing a DOS attack. 1. http://www.webhostingtalk.com/showthread.php?t=1285835 http://www.webhostingtalk.com/showthread.php?t=1285835

pilif 13y ago

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them). When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data. Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.

esw 13y ago
This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.
- pilif 13y ago
  I would assume that sites affected by a Network Solutions downtime would qualify as small(ish) sites because the big ones have other solutions anyways. Also, it's much easier to deal with a DOS on DNS than with a DOS on your app servers. If you can handle it for your app servers, you'll be able to handle if for DNS. If not, well, then you're down anyways.
- dsl 13y ago
  FYI, Zerigo DNS is a joke. They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707 https://news.ycombinator.com/item?id=5922707
13y ago

asr2bd 13y ago

Our site has been sporadically up and down for the past few hours. Glad I know the culprit finally. Hope this gets resolved soon

nodata 13y ago

Site down? Don't link to it. Bad netiquette. Jesus christ.

TranceMan 13y ago

Their facebook page has been updated: Network Solutions is experiencing a Distributed Denial of Service (DDOS) attack that is impacting our customers as well as the Network Solutions site. Our technology team is working to mitigate the situation. Please check back for updates. https://www.facebook.com/networksolutions/posts/10151468105866020 https://www.facebook.com/networksolutions/posts/101514681058...

ottoflux 13y ago

Back up.

SubZero 13y ago

Why did I click on the link to a down website? What did I expect? Shame on me.

Tenhundfeld 13y ago

TechZone360 (ugh, so many ads) article with a little more information: Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/17/345902-network-solutions-experiences-hijacking-dns-records.htm http://www.techzone360.com/topics/techzone/articles/2013/07/...

muraiki 13y ago
A comment in the article pointed out that this one seems very similar to a Cisco article from a month ago: http://blogs.cisco.com/security/hijacking-of-dns-records-from-network-solutions/ http://blogs.cisco.com/security/hijacking-of-dns-records-fro...

dangayle 13y ago

Just what I wanted to wake up to. Client emails asking me why their sites were down. Sigh.

dholowiski 13y ago

This appears to be a problem again this afternoon (1:44PM MST)