11 ms·

As a sysadmin with 20+ years experience, I've had long TTLs cause issues on several occasions. I've never regretted a short TTL.

tyingq 6y ago
I would put a reasonable floor on short TTLs (5 minutes?), but yes, it's nice in an emergency to be able to send everyone someplace new. Sucks if you're down, hacked, etc, and can't do anything about an existing long TTL other than wait it out. Edit: Worth noting there's lots of software that seems to only resolve hostnames at first connection, then hangs onto it forever. Lots of java internals for example, unless you poke in specific configuration.
- ses1984 6y ago
  Nginx as reverse proxy does this and it's burned me.
jamescun 6y ago
This mirrors my experience. I've seen more problems caused by the JVM, by default on some configurations, caching DNS indefinitely, regardless of TTL, than caused by a short TTL.
- icedchai 6y ago
  Definitely. About 12+ years ago, I had to prove to a vendor, with tcpdumps, etc., that they were connecting to the wrong server after we changed a DNS entry. 3 of their systems were working, the 4th hadn't been restarted and was connecting to the old address. Very frustrating.
- axaxs 6y ago
  Is that fixed yet? I remember having to bounce Java apps every time DNS changed, which never made sense to me. It's literally the point of DNS to not have to do that.
  - icedchai 6y ago
    It's fixed in newer JVMs ("newer" meaning anything in the past 10 years.)
- squiggleblaz 6y ago
  But that isn't the issue at question. If the TTL is 300 seconds or 3600 seconds and the JVM holds onto it for three weeks, you can't blame a TTL of 3600 seconds for that, and setting the TTL down to 0 seconds at all isn't going to fix it either (unless, bizarrely, the JVM developers decided to respect TTLs of 0-60 and treat all other values as infinite).
_joel 6y ago
Agreed, I have been stung by long propogation times, never shorter times to propogate.
jabart 6y ago
Same, saw a DNS entry for an internal DB endpoint get updated with an 8 hour TTL for a planned failover. DB Admins went to sleep as everything was fine, everyone woke up 8 hours later with everything failing to connect. Had to flush the DNS on each internal server to fix that issue.
- cookiengineer 6y ago
  The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.
  - rhacker 6y ago
    I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly static forever. We don't live in the old world where we had to take a server down any more. It's all managed.
    - laurent92 6y ago
      What about 2 minutes during the first hour after a change, 30 minutes during 2 days, 3 hrs after that?
      - squiggleblaz 6y ago
        I think the point is that I don't want to change my TTLs. If I had my druthers, yeah, they'd go up periodically, and then if tech support receives an issue that sounds like it might require DNS changes, they press a button and the TTL drops. Once that issue is closed, it goes back to gradually increasing.
ransom1538 6y ago
"OH, thank god it's a 48 hour TTL". No one ever said this.
- notriddle 6y ago
  48hrs is excessively long. Most DNS servers will probably evict your entry from the cache before two days elapse anyway. Less than a minute is excessively short. Most long-lived applications will internally cache your IP address for longer than a minute no matter what you do.
dgellow 6y ago
Yep, that matches perfectly my (shorter) experience
vfc1 6y ago
I've been using 5 minute TTL in production for years, never noticed any problems with it. It has the advantage that it makes it super simple to deploy to production at a moment's notice, in case something unexpected occurs.
deleted 6y ago
[deleted]
arp242 6y ago
The part that's missing in DNS is the ability to tell people that the cache expired. You can't really do this in DNS itself since a cached response will never hit the network. You could add a signal in the application protocol for example: <meta name="dns-refresh" value="[.. some date ..]">. You'd also have to do that for SMTP, IMAP, etc. Probably not worth the complexity as low TTLs seem to work well enough with few enough downsides. DNS is already a tricky enough protocol.
- pas 6y ago
  Use a new (sub)domain. Your main domain can then simply redirect to whatever subdomain you wish.
  - ProtoAES256 6y ago
    A new subdomain normally will do the trick, but might be suboptimal due to the work needed to be done according to scale/tech/job involved. For example, doing this as a sys in a big company requires jumping through bureaucracy hoops, not to mention setting up ssl for it(I know there is letsencrypt but you know, some coporate doesn't want them). A short TTL will just be "Well just try it again in 10 minutes and it'll work."
    - selcuka 6y ago
      Besides we might still want other caches (such as the browser cache) to work. A new subdomain would invalidate those too.
- csunbird 6y ago
  (offline) Cache invalidation is a very hard thing to do. If you are going to ask the network if your DNS cache is still valid, you might as well get the latest value anyways, since DNS queries are tiny.
- znpy 6y ago
  Not all the world is HTTP and/or web stuff. For a lot of things changing domain is quite a hassle, and the old domain will still be around, cached, and considered valid (because its ttl hasn't expired yet). I agree with GP in saying that a short TTL never created too much issue. Now, to come to the original issue: short ttl. What do you expect, in this day and age? With all this cloud stuff going up and down, being created and destroyed again and again, ephemeral stuff that doesn't even last a whole single day... Of course you need a short ttl.
is_true 6y ago
I always go for 5 minutes. The workaround would be to lower the TTL we you know you are gonna make changes soon.
marcosdumay 6y ago
> I've never regretted a short TTL. I've seen enough people complaining about overloaded DNS servers.
- sparrish 6y ago
  Yes, that happens. I've seen way more people complain about outages that lasted hours because of long TTLs and a deployment mistake.
- znpy 6y ago
  I haven't.
_wldu 6y ago
15 mins is the shortest I would want to go. Otherwise, your app is going to be perceived as slow (by your users) because it has to do a bunch of needless DNS lookups a lot.
- hn_throwaway_99 6y ago
  I doubt the perception in slowness is really going to be that different between a 5 min TTL and a 15 min TTL. 1. DNS lookups add on the order of 100ms to load time. 2. In both cases (5 min and 15 min) the user is going to do a DNS lookup on the first page, then have cached DNS while they browse for a bit, then have a DNS lookup at some point in the future. Most web sessions are relatively short, so I doubt most users would even notice the difference where they see an extra 100ms load time at, say, 2 points in their session instead of 1. In my experience the appropriate use of the preconnect and dns-prefetch hints has a much bigger impact on perceived performance than worrying about DNS TTLs beyond 5 min.
KaiserPro 6y ago
> I've never regretted a short TTL. Very much this sentiment. When migrating a website many years ago, I forgot to lower the TTL of companyname.co.uk. I had lowered the TTL of www. but not the root. So when the big switchover came, half the traffic stayed where it was. Not only that the new backend fell over. having that option to roll out/in would have been really useful. Now with the cloud stuff goes away with, so having a TTL of 3600 means long outages.
1vuio0pswjnm7 6y ago
User perspective: As a user with 20+ years of experience, I have never regretted using authoritative DNS as the source for DNS data instead of DNS caches. Instead of using a shared cache, I bind local authoritative servers loaded with zone files containing the DNS data that applications need to the loopback. IME, lookups are faster than with a cache. IMO as a user, recursive queries to shared caches are overrated. I want to know what lookups the applications I use are making and I want control over what DNS data is available to them. IME, giving applications license to lookup any resource at any time means this permission will be abused to further the interests of the online ad industry and all those who service it instead of 100% serving the interests of the user. For example, a commercial router's management console does not need to be able to look up the addresses of ad servers. Letting it have access to a remote shared cache, or even a local one, that does recursive queries is unnecessary. The user has already paid for the router.

tyingq 6y ago

I would put a reasonable floor on short TTLs (5 minutes?), but yes, it's nice in an emergency to be able to send everyone someplace new. Sucks if you're down, hacked, etc, and can't do anything about an existing long TTL other than wait it out. Edit: Worth noting there's lots of software that seems to only resolve hostnames at first connection, then hangs onto it forever. Lots of java internals for example, unless you poke in specific configuration.

ses1984 6y ago
Nginx as reverse proxy does this and it's burned me.

jamescun 6y ago

This mirrors my experience. I've seen more problems caused by the JVM, by default on some configurations, caching DNS indefinitely, regardless of TTL, than caused by a short TTL.

icedchai 6y ago
Definitely. About 12+ years ago, I had to prove to a vendor, with tcpdumps, etc., that they were connecting to the wrong server after we changed a DNS entry. 3 of their systems were working, the 4th hadn't been restarted and was connecting to the old address. Very frustrating.
axaxs 6y ago
Is that fixed yet? I remember having to bounce Java apps every time DNS changed, which never made sense to me. It's literally the point of DNS to not have to do that.
- icedchai 6y ago
  It's fixed in newer JVMs ("newer" meaning anything in the past 10 years.)
squiggleblaz 6y ago
But that isn't the issue at question. If the TTL is 300 seconds or 3600 seconds and the JVM holds onto it for three weeks, you can't blame a TTL of 3600 seconds for that, and setting the TTL down to 0 seconds at all isn't going to fix it either (unless, bizarrely, the JVM developers decided to respect TTLs of 0-60 and treat all other values as infinite).

_joel 6y ago

Agreed, I have been stung by long propogation times, never shorter times to propogate.

jabart 6y ago

Same, saw a DNS entry for an internal DB endpoint get updated with an 8 hour TTL for a planned failover. DB Admins went to sleep as everything was fine, everyone woke up 8 hours later with everything failing to connect. Had to flush the DNS on each internal server to fix that issue.

cookiengineer 6y ago
The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.
- rhacker 6y ago
  I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly static forever. We don't live in the old world where we had to take a server down any more. It's all managed.
  - laurent92 6y ago
    What about 2 minutes during the first hour after a change, 30 minutes during 2 days, 3 hrs after that?
    - squiggleblaz 6y ago
      I think the point is that I don't want to change my TTLs. If I had my druthers, yeah, they'd go up periodically, and then if tech support receives an issue that sounds like it might require DNS changes, they press a button and the TTL drops. Once that issue is closed, it goes back to gradually increasing.

ransom1538 6y ago

"OH, thank god it's a 48 hour TTL". No one ever said this.

notriddle 6y ago
48hrs is excessively long. Most DNS servers will probably evict your entry from the cache before two days elapse anyway. Less than a minute is excessively short. Most long-lived applications will internally cache your IP address for longer than a minute no matter what you do.

dgellow 6y ago

Yep, that matches perfectly my (shorter) experience

vfc1 6y ago

I've been using 5 minute TTL in production for years, never noticed any problems with it. It has the advantage that it makes it super simple to deploy to production at a moment's notice, in case something unexpected occurs.

deleted 6y ago

[deleted]

arp242 6y ago

The part that's missing in DNS is the ability to tell people that the cache expired. You can't really do this in DNS itself since a cached response will never hit the network. You could add a signal in the application protocol for example: <meta name="dns-refresh" value="[.. some date ..]">. You'd also have to do that for SMTP, IMAP, etc. Probably not worth the complexity as low TTLs seem to work well enough with few enough downsides. DNS is already a tricky enough protocol.

pas 6y ago
Use a new (sub)domain. Your main domain can then simply redirect to whatever subdomain you wish.
- ProtoAES256 6y ago
  A new subdomain normally will do the trick, but might be suboptimal due to the work needed to be done according to scale/tech/job involved. For example, doing this as a sys in a big company requires jumping through bureaucracy hoops, not to mention setting up ssl for it(I know there is letsencrypt but you know, some coporate doesn't want them). A short TTL will just be "Well just try it again in 10 minutes and it'll work."
  - selcuka 6y ago
    Besides we might still want other caches (such as the browser cache) to work. A new subdomain would invalidate those too.
csunbird 6y ago
(offline) Cache invalidation is a very hard thing to do. If you are going to ask the network if your DNS cache is still valid, you might as well get the latest value anyways, since DNS queries are tiny.
znpy 6y ago
Not all the world is HTTP and/or web stuff. For a lot of things changing domain is quite a hassle, and the old domain will still be around, cached, and considered valid (because its ttl hasn't expired yet). I agree with GP in saying that a short TTL never created too much issue. Now, to come to the original issue: short ttl. What do you expect, in this day and age? With all this cloud stuff going up and down, being created and destroyed again and again, ephemeral stuff that doesn't even last a whole single day... Of course you need a short ttl.

is_true 6y ago

I always go for 5 minutes. The workaround would be to lower the TTL we you know you are gonna make changes soon.

marcosdumay 6y ago

> I've never regretted a short TTL. I've seen enough people complaining about overloaded DNS servers.

sparrish 6y ago
Yes, that happens. I've seen way more people complain about outages that lasted hours because of long TTLs and a deployment mistake.
znpy 6y ago
I haven't.

_wldu 6y ago

15 mins is the shortest I would want to go. Otherwise, your app is going to be perceived as slow (by your users) because it has to do a bunch of needless DNS lookups a lot.

hn_throwaway_99 6y ago
I doubt the perception in slowness is really going to be that different between a 5 min TTL and a 15 min TTL. 1. DNS lookups add on the order of 100ms to load time. 2. In both cases (5 min and 15 min) the user is going to do a DNS lookup on the first page, then have cached DNS while they browse for a bit, then have a DNS lookup at some point in the future. Most web sessions are relatively short, so I doubt most users would even notice the difference where they see an extra 100ms load time at, say, 2 points in their session instead of 1. In my experience the appropriate use of the preconnect and dns-prefetch hints has a much bigger impact on perceived performance than worrying about DNS TTLs beyond 5 min.

KaiserPro 6y ago

> I've never regretted a short TTL. Very much this sentiment. When migrating a website many years ago, I forgot to lower the TTL of companyname.co.uk. I had lowered the TTL of www. but not the root. So when the big switchover came, half the traffic stayed where it was. Not only that the new backend fell over. having that option to roll out/in would have been really useful. Now with the cloud stuff goes away with, so having a TTL of 3600 means long outages.

1vuio0pswjnm7 6y ago

User perspective: As a user with 20+ years of experience, I have never regretted using authoritative DNS as the source for DNS data instead of DNS caches. Instead of using a shared cache, I bind local authoritative servers loaded with zone files containing the DNS data that applications need to the loopback. IME, lookups are faster than with a cache. IMO as a user, recursive queries to shared caches are overrated. I want to know what lookups the applications I use are making and I want control over what DNS data is available to them. IME, giving applications license to lookup any resource at any time means this permission will be abused to further the interests of the online ad industry and all those who service it instead of 100% serving the interests of the user. For example, a commercial router's management console does not need to be able to look up the addresses of ad servers. Letting it have access to a remote shared cache, or even a local one, that does recursive queries is unnecessary. The user has already paid for the router.