3 ms·

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users through flushing their local DNS cache so the users can get back to work before TTL expires on a broken entry. The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. Most sites I've worked with change DNS <1 time per year, so outside infrequent, planned maintenance a long cache time is best for performance.

donmcronald 4y ago
> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. This is my preferred way of doing it, especially for MX records.
- jve 4y ago
  Or you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.
  - eatbitseveryday 4y ago
    That’s exactly what was said in the above comments
Tepix 4y ago
Quoting from https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html : Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.
- dspillett 4y ago
  I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem. I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs (<300s) as errors and "corrected" them by using its default (often 24 hours). I actually encountered it and its effect in the wild, but this was back in the very late 90s or very early 00s. It was a DNS cache daemon on an old Unix variant (I forget which one, my university had a few floating around at the time), so old that if someone is still using it they have far greater issues than my low TTL value! My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short enough for anything I envisage doing ATM & 300s seems to be reliable) and if someone's DNS cache is broken that is their problem, much like I don't pay any attention to issues Internet Explorer users might have looking at some HTML+CSS+JS I've inexpertly strung together.
  - pastage 4y ago
    Getting DNS right is non trivial, and UDP will be dropped at times. If you have to lookup the name every two seconds means you have a high risk of getting failed lookups even if your infra team is competent. (10 minutes should be ok)
  - LinuxBender 4y ago
    I wonder what those clients are Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.
advisedwang 4y ago
If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching) Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).
- 8organicbits 4y ago
  Testing strategy is also pretty important here. If you can run integration tests that bypass DNS caches immediately after the change, then you should know quick if things are working. If both old and new DNS settings are valid, the slow onset works great and minimizes impact of an error. I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.
- M3L0NM4N 4y ago
  I think the amount of people that would see the error either way would be the same, assuming constant fix time and traffic. This means lowering the TTL as much as possible should be the right solution, because those affected will only have the issue for as long as the fix takes (hopefully less than 1hr) whereas if it is kept at 1hr those affected by the error will have it for an entire hour. Disclaimer I'm not OP

donmcronald 4y ago

> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. This is my preferred way of doing it, especially for MX records.

jve 4y ago
Or you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.
- eatbitseveryday 4y ago
  That’s exactly what was said in the above comments

Tepix 4y ago

Quoting from https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html : Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.

dspillett 4y ago
I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem. I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs (<300s) as errors and "corrected" them by using its default (often 24 hours). I actually encountered it and its effect in the wild, but this was back in the very late 90s or very early 00s. It was a DNS cache daemon on an old Unix variant (I forget which one, my university had a few floating around at the time), so old that if someone is still using it they have far greater issues than my low TTL value! My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short enough for anything I envisage doing ATM & 300s seems to be reliable) and if someone's DNS cache is broken that is their problem, much like I don't pay any attention to issues Internet Explorer users might have looking at some HTML+CSS+JS I've inexpertly strung together.
- pastage 4y ago
  Getting DNS right is non trivial, and UDP will be dropped at times. If you have to lookup the name every two seconds means you have a high risk of getting failed lookups even if your infra team is competent. (10 minutes should be ok)
- LinuxBender 4y ago
  I wonder what those clients are Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.

advisedwang 4y ago

If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching) Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).

8organicbits 4y ago
Testing strategy is also pretty important here. If you can run integration tests that bypass DNS caches immediately after the change, then you should know quick if things are working. If both old and new DNS settings are valid, the slow onset works great and minimizes impact of an error. I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.
M3L0NM4N 4y ago
I think the amount of people that would see the error either way would be the same, assuming constant fix time and traffic. This means lowering the TTL as much as possible should be the right solution, because those affected will only have the issue for as long as the fix takes (hopefully less than 1hr) whereas if it is kept at 1hr those affected by the error will have it for an entire hour. Disclaimer I'm not OP