5 ms·

Let's Make DNS Outage Suck Less

37 points by kvz 14y ago

trotsky 14y ago
dnsmasq --interface=lo --all-servers --server=172.16.0.23 --server=8.8.8.8 --server=4.2.2.2 echo "nameserver 127.0.0.1" > /etc/resolv.conf http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html --all-servers : By default, when dnsmasq has more than one upstream server available, it will send queries to just one server. Setting this flag forces dnsmasq to send all queries to all available servers. The reply from the server which answers first will be returned to the original requester. Dnsmasq also has avoidance of unresponsive servers built in, this is a bit more of a blunt instrument.
- kvz 14y ago
  Thanks for the link! I did find dnrd (http://dnrd.sourceforge.net/ http://dnrd.sourceforge.net/), but I really wasn't looking into proxying DNS requests and introducing another daemon / dependency, that could at some point fail or get confused, requiring even more tech to keep it alive, or provide insight in it's workings/reasoning. Really wanted to solve this using as transparent & archaic tech as possible in the hope this still stands when big player's resolving servers fall. Granted, it is blunt; archaic tools often are ; )
- sigil 14y ago
  The example you gave won't stick for most people: echo "nameserver 127.0.0.1" > /etc/resolv.conf If you use dhcp3-client (chances are you do), add this line to /etc/dhcp/dhcplient.conf to try your local DNS cache first: prepend domain-name-servers 127.0.0.1; Note that this will slow down stuff like captive portals at airports that want to push you to their "accept terms" page first. I use djb's dnscache instead of dnsmasq, but dnsmasq works fine too. http://cr.yp.to/djbdns/run-cache.html http://cr.yp.to/djbdns/run-cache.html
jefe78 14y ago
They're using DNS for DB communication. That's a major faux pas.
- kvz 14y ago
  Using IPs for communication to Amazon RDS instances is inadvisable. You'd have to use WAN addresses outside of EC2, and LAN addresses inside of EC2. Addressing them by domain resolves this. Additionally I don't think Amazon guarantees your box will be accessible on the same IP addresses after Multi-AZ failovers.
  - jefe78 14y ago
    You're definitely doing it wrong.
datums 14y ago
If you use the rotate option with timeout you'll quickly hop to the next working resolver. http://edwin.io/optimized-resolv-conf http://edwin.io/optimized-resolv-conf Do you have data you can share regarding "Amazon EC2 resolving nameserver (172.16.0.23) is unreachable too often" ?
- dsr_ 14y ago
  You don't need to do rotation, either -- suppose the first nameserver, when up, is consistently faster than the others. Just setting a timeout of 1 is appropriate, then.
  - kvz 14y ago
    A timeout of 1 can result in false positives. 3 seconds is recommended according to the manpage. Should the occasional false positive be none of your concern, then you still add at least a second to every request. If you make many, that's many seconds.
  - datums 14y ago
    This is more of a performance optimization and less about a HA solution. With rotation you also avoid being throttled. If that's not a concern based on the # of request you make then you can simply timeout and go to the next resolver on the list.
- kvz 14y ago
  From the manpage: "[rotate] causes round robin selection of nameservers from among those listed". This means in your configuration, the broken nameserver will be hit once every 3 times, resulting in a 1second timeout before trying the other 2. So in every 9 requests your setup results in 3 seconds delay for as long as the nameserver is down. In case of Amazon, you actually prefer their nameserver as it resolves names to LAN IPs where possible. With your strategy, ~66% of my instance hostnames would be resolved to the less efficient WAN IPs, that's with all 3 nameservers being reachable. As for data, there's papertrail and Premium Support tickets with Amazon admitting to their downtimes. In most cases (e.g. the one on Jan 14, 2013 11:27 PM PST) "there was a problem with the underlying host". Remember it doesn't even need to be the service itself going down, there's weaker links in this chain like misconfiguration of the virtual host. Cables can get tripped over. Anycast is not possible here. I'm not blaming them. This article was about mitigating suckyness in case of outages, not proving that such a thing even exists : )
3amOpsGuy 14y ago
You know, sometimes the simpler solutions work best. I like this one. I'm sitting here contrasting this against my normal approach* and the approach here is: 1) easier to explain to others 2) self documenting 3) just as effective as running a caching nameserver * i have vm hosts configured in CM to have a bunch of promises applied, one of them is to run a caching nameserver, these hosts are the only ones allowed to do zone transfers. The vm instances running on top of these have a promise applied which has them use their underlying dom0 for dns queries.

37 points by kvz 14y ago

dnsmasq --interface=lo --all-servers --server=172.16.0.23 --server=8.8.8.8 --server=4.2.2.2 echo "nameserver 127.0.0.1" > /etc/resolv.conf http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html --all-servers : By default, when dnsmasq has more than one upstream server available, it will send queries to just one server. Setting this flag forces dnsmasq to send all queries to all available servers. The reply from the server which answers first will be returned to the original requester. Dnsmasq also has avoidance of unresponsive servers built in, this is a bit more of a blunt instrument.

kvz 14y ago
Thanks for the link! I did find dnrd (http://dnrd.sourceforge.net/ http://dnrd.sourceforge.net/), but I really wasn't looking into proxying DNS requests and introducing another daemon / dependency, that could at some point fail or get confused, requiring even more tech to keep it alive, or provide insight in it's workings/reasoning. Really wanted to solve this using as transparent & archaic tech as possible in the hope this still stands when big player's resolving servers fall. Granted, it is blunt; archaic tools often are ; )
sigil 14y ago
The example you gave won't stick for most people: echo "nameserver 127.0.0.1" > /etc/resolv.conf If you use dhcp3-client (chances are you do), add this line to /etc/dhcp/dhcplient.conf to try your local DNS cache first: prepend domain-name-servers 127.0.0.1; Note that this will slow down stuff like captive portals at airports that want to push you to their "accept terms" page first. I use djb's dnscache instead of dnsmasq, but dnsmasq works fine too. http://cr.yp.to/djbdns/run-cache.html http://cr.yp.to/djbdns/run-cache.html

jefe78 14y ago

They're using DNS for DB communication. That's a major faux pas.

kvz 14y ago
Using IPs for communication to Amazon RDS instances is inadvisable. You'd have to use WAN addresses outside of EC2, and LAN addresses inside of EC2. Addressing them by domain resolves this. Additionally I don't think Amazon guarantees your box will be accessible on the same IP addresses after Multi-AZ failovers.
- jefe78 14y ago
  You're definitely doing it wrong.

datums 14y ago

If you use the rotate option with timeout you'll quickly hop to the next working resolver. http://edwin.io/optimized-resolv-conf http://edwin.io/optimized-resolv-conf Do you have data you can share regarding "Amazon EC2 resolving nameserver (172.16.0.23) is unreachable too often" ?

dsr_ 14y ago
You don't need to do rotation, either -- suppose the first nameserver, when up, is consistently faster than the others. Just setting a timeout of 1 is appropriate, then.
- kvz 14y ago
  A timeout of 1 can result in false positives. 3 seconds is recommended according to the manpage. Should the occasional false positive be none of your concern, then you still add at least a second to every request. If you make many, that's many seconds.
- datums 14y ago
  This is more of a performance optimization and less about a HA solution. With rotation you also avoid being throttled. If that's not a concern based on the # of request you make then you can simply timeout and go to the next resolver on the list.
kvz 14y ago
From the manpage: "[rotate] causes round robin selection of nameservers from among those listed". This means in your configuration, the broken nameserver will be hit once every 3 times, resulting in a 1second timeout before trying the other 2. So in every 9 requests your setup results in 3 seconds delay for as long as the nameserver is down. In case of Amazon, you actually prefer their nameserver as it resolves names to LAN IPs where possible. With your strategy, ~66% of my instance hostnames would be resolved to the less efficient WAN IPs, that's with all 3 nameservers being reachable. As for data, there's papertrail and Premium Support tickets with Amazon admitting to their downtimes. In most cases (e.g. the one on Jan 14, 2013 11:27 PM PST) "there was a problem with the underlying host". Remember it doesn't even need to be the service itself going down, there's weaker links in this chain like misconfiguration of the virtual host. Cables can get tripped over. Anycast is not possible here. I'm not blaming them. This article was about mitigating suckyness in case of outages, not proving that such a thing even exists : )

3amOpsGuy 14y ago

You know, sometimes the simpler solutions work best. I like this one. I'm sitting here contrasting this against my normal approach* and the approach here is: 1) easier to explain to others 2) self documenting 3) just as effective as running a caching nameserver * i have vm hosts configured in CM to have a bunch of promises applied, one of them is to run a caching nameserver, these hosts are the only ones allowed to do zone transfers. The vm instances running on top of these have a promise applied which has them use their underlying dom0 for dns queries.