10 ms·
Stop using low DNS TTLs (2019)
- necovek 4y agoI interpret their data differently: only 47% of DNS lookups were for expired entries due to TTL. If some of those were for longish TTLs, some for actually changed records, then an even bigger majority of requests would be needed anyway, and longer TTLs would not have stopped them.
- antonvs 4y ago> DNS caching has become next to useless. Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.
- blueflow 4y agoIt made as much sense as it does now, but differently from back then, people today neglect the performance and reliability impact it has.
- sofixa 4y agoNot really, because infrastructure is much more dynamic today. Maybe if your IPs never changed long TTLs are OK, but when you're running behind a cloud load balancer, with changing IPs and failover to different regions/providers, low TTLs make more sense.
- blueflow 4y agoIts understandable that you need to change your routing dynamically, but DNS is not the right tool for that.
- solatic 4y agoAWS Route53 charges per-query. At scale, using longer TTLs, if clients respect the longer TTLs, has a measurable impact on your bill.
- dexterdog 4y agoThat's why you use aliases as much as possible since they are not charged.
- jesprenj 4y agoIt makes a lot of sense, if we think about how the protocol works. Without caching, I assume every client would send every query in it's entirety to the root server, right? Since a single zone can have multiple labels ({abc.def}.ghi.example - abc.def is a NAME in ghi.example), the server needs to receive the entire requested name in order to properly respond (yeah, root queries usualy optimize this, because all NAMEs there are one label long).
- speedgoose 4y agoI use low TTLs because I don’t like waiting too long when I broke something or did a DNS configuration mistake.
- danjc 4y agoOne reason to use low TTL’s is for a DNS-based load balancer that is typically used for DR such as Azure Traffic Manager. That and paranoia about incorrectly updating a record causing your service to break for a period of time with nothing you can do about it.
- deleted 4y ago[deleted]
- rgj 4y agoThe way of measuring this affects (and skews) the outcome. DNS queries with low TTLs are requested more frequently and because of that, the author seeing more of them pass through his patched DNS relay, which was only left running for a few hours instead of for at least the max TTL they wanted to measure.
- jesprenj 4y agoThat's what I thought of as well. Data can be presented in many ways to show many meanings at first sight. If charts would be generated in number of unique NAMEs in relation to TTL, readers would probably get a diffetent feeling. Or maybe not. I don't know, but this issue probably has a name ... selection bias?
- wink 4y agoYeah,that was one of my first thoughts as well. I guess the raw.github example gets not only more traffic (irrelevant here) but probably also many more requests than hundreds of other sites combined. Also no clue how actual RR DNS load balancing (without anycast) is still widely practiced, because I guess that also influences it a lot, you may be able to fix a DC going down on a completely different level than DNS. And my experience with cheap home routers is basically that they might cache hostnames for a month (we had stuff with tracking pixels in 2013-2017, EOLing some stuff was wild, we switched DNS and days and weeksl later still got valid requests to the old IPs).
- 8organicbits 4y agoThe scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users through flushing their local DNS cache so the users can get back to work before TTL expires on a broken entry. The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. Most sites I've worked with change DNS <1 time per year, so outside infrequent, planned maintenance a long cache time is best for performance.
- donmcronald 4y ago> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. This is my preferred way of doing it, especially for MX records.
- jve 4y agoOr you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.
- eatbitseveryday 4y agoThat’s exactly what was said in the above comments
- Tepix 4y agoQuoting from https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html : Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.
- theamk 4y agoNote this data many no longer be correct. I just checked and numbers for some of the websites are much longer now: raw.githubusercontent.com has 1 hour TTL. github.map.fastly.net is 5 minutes TTL. detectportal.firefox.com is 60/120 seconds TTL. Another thing missing from the post is exactly _how_ bad is it -- because latency matters. If there is a choice of 30mS extra latency on initial site visit vs a chance of 1 hour of downtime in case of hardware failure, it is not clear what the best choice is. Especially things like detectportal.firefox.com which are asynchronous and normally do not introduce extra user-visible latency at all.
- cmeacham98 4y agoMost clients ignore TTLs anyways, so your 1 day TTL barely is helping you're website's performance at all: https://www.ctrl.blog/entry/dns-client-ttl.html https://www.ctrl.blog/entry/dns-client-ttl.html
- ikiris 4y agoAnd yet most of that doesn't matter, because you almost always have local resolvers that do cache properly.
- cmeacham98 4y agoYour average consumer router is a pile of garbage that barely works. I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it. Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.
- bagels 4y agoConsumer router doesn't seem to need to given that they typically have such low client counts. Your computer is doing caching, so there's not much benefit for your home router to also do caching.
- ikiris 4y agoThe latency to most ISP cores is low ms range. You really should just stop digging.
- cmeacham98 4y agoMaybe you live in a country where ISP routers are well designed and low latency, but where I live in the US the primary latency in DNS lookup is almost always the round trip to the horribly configured and slow ISP DNS.
- jedberg 4y agoDNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48. It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice. [0] https://www.ctrl.blog/entry/dns-client-ttl.html https://www.ctrl.blog/entry/dns-client-ttl.html
- toast0 4y agoI'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not. If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?
- deleted 4y ago[deleted]
- jedberg 4y agoThat's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.
- donavanm 4y agoFrom the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).
- iopq 4y agoI like changing my IP address from time to time. I host my own VPN and connect to it through a domain. When I update my IP, I have no internet access for whatever the TTL length is.
- arpa 4y agoWell if you don't like people using low TTLs maybe you should have put validations in the original spec. Oh you didn't write the spec? well sucks to be you then.
- outofmyshed 4y agoAnyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with. I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.
- mrb 4y agoIf your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing. Unless your traffic is non-revenue generating or somehow poorly monetizable...
- withinboredom 4y ago> So a few thousand dollars is nothing. In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.
- akira2501 4y ago> a few thousands in UNEXPECTED costs, is huge. It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month? It's a fixed _overhead_ issue, not a fixed _budget_ issue.
- withinboredom 4y agoYou should be reserving those instances so it IS a FIXed cost, with very few exceptions.
- 4y ago
- charcircuit 4y ago>The urban legend that DNS-based load balancing depends on TTLs (it doesn’t). It does. You need to get loaded servers out of what's being advertised. The only way to do this is with a TTL.
- notpushkin 4y agoI'm using Netlify DNS for most of my websites. Usually I set TTL low (~600 seconds) in order to quickly change configuration in case I fuck up. When things are working though, I remember that Netlify doesn't have an edit function for DNS, so I just leave it as is.
- Tepix 4y agoOne cool thing about DJBs tinydns is that you can configure TTLs if you plan to roll over to another system at a particular point in time. I.e. you can have multiple records each with its own time range where it is valid. From the tinydns-data documentation: -- You may include a timestamp on each line. If ttl is nonzero (or omitted), the timestamp is a starting time for the information in the line; the line will be ignored before that time. If ttl is zero, the timestamp is an ending time (``time to die'') for the information in the line; tinydns dynamically adjusts ttl so that the line's DNS records are not cached for more than a few seconds past the ending time. A timestamp is an external TAI64 timestamp, printed as 16 lowercase hexadecimal characters. For example, the lines +www.heaven.af.mil:1.2.3.4:0:4000000038af1379 +www.heaven.af.mil:1.2.3.7::4000000038af1379 specify that www.heaven.af.mil will have address 1.2.3.4 until time 4000000038af1379 (2000-02-19 22:04:31 UTC) and will then switch to IP address 1.2.3.7. -- Do other authorative resolvers support this, too?
- vinay_ys 4y agoYeah, this works if it is a planned cutover. If you are having to ditch your current IP network in a hurry, then this approach won't work.
- hayst4ck 4y agoAn outage at a large (but not mega) company will likely have a 5-10 minute period to get looked at, a 5-10 minute period to understand the problem, and a 5-10 minute period for resolution. Any DNS related problem with a TTL higher than 5-10 minutes could greatly extend an outage.
- msravi 4y agoIf you use a pihole, you can force the minimum ttl for downstream clients (upto a maximum value of 3600s) by setting, for example, min-cache-ttl=2400 in /etc/dnsmasq.d/ Never had a problem with this. Of course, you can force a flush of the cache using `pihole restartdns` in case you need to.
- bagels 4y agoIs there a push mechanism in DNS? My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs. (I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)
- School-Cotton 4y agoHow would that even work, since you can’t in general know who has cached the entries you control? Broadcast updates to the whole internet?
- wnoise 4y agoThere is the NOTIFY extension, but it's for master servers to notify secondary servers that they should do a pull. Not really usable for unknown general clients, of course.
- wahern 4y agoThere's DNS NOTIFY (RFC 1996), which is commonly used to trigger a pull. But to minimize malfeasance it's usually ignored unless authenticated (e.g. via DNS TSIG) or appears to originate from a known address. The typical scenario is a master nameserver notifying a list of [known] slaves about a zone update. There's also DNS Push (RFC 8765), but it's a very recent mechanism. I'm not familiar, but I doubt it's widely supported.
- vbezhenar 4y agoI almost always use 24 hours DNS TTL except for things like dynamic DNS. I never felt the need to urgently change DNS.
- thethetermin 4y agoAnother reason to set a low DNS TTL, is if you are running a service with very limited resources and still want some kind of HA / Disaster recovery. For instance you setup your web app in both Digital Ocean and Linode, with a system that takes the database (for instance mysql replication) and the web app synchronized in the "replica" VM. When something bad happens and you can't recover, you switch DNS and that's it.
- wink 4y agoFascinating read. Of course I'm no big business but for the last 20 years I've done it like this for non-critical stuff: - NS Records at 86400 - A and CNAME at 3600 - 1-2 days before moving stuff to other infra (ISP, DNS reseller, etc), switch to 600 or lower, then 1-2 days later switch back. Can't even say why, it's cargo culting but it has always worked out fine, but ofc I'm not talking about production APIs. And every single time there was some problem, setting the TTL to < 1minute would not have helped, so I guess I'll stick to it.
- nijave 4y agoIt seems the only reason given is "some clients are broken"? They mention latency but I wonder what the real world impact is. On a home network, most users are using ISP supplied DNS which /usually/ has pretty low latency and will have a huge cache already built up. Corporate users will have DNS cached to all the common host they're visiting. Anecdotally, AWS ALBs (HTTP load balancers) have a CNAME with a 10 minute TTL (maybe it is 60 seconds) and can have their IPs change 100s of times a month and this works fine as a default configuration for many use cases.
- megiddo 4y agoIME DNS TTLs are low because debugging DNS during development is a pain. So you set them low the first time you run into a DNS issue where you had to wait an hour or a day to resolve some bad configuration and your "fuck this" meter went from 0 to 100.
- mschuster91 4y agoI've been bitten in the rear far too often with high TTLs. Especially stuff like AWS Beanstalk which tends to delete and recreate resources if you're not extremely careful... thanks but I'll stay with 60s or 300s TTLs.
- emaildanwilson 4y agoEven if the DNS update is available to an end user, if you're using HTTP2 then web browsers will keep the connection to the old IP address until they receive a GOAWAY frame.
- sylware 4y agoOoops! good point, I should check the DNS TTL for my shelf-hosted email server.
- technovader 4y agoThere are use cases for low TTLs and use cases for high TTLs The statement here is misguided. Use whatever TTL serves your purpose. DNS was designed to be distributed and scalable. And in my opinion it is THE MOST scalable protocol as it basically allowed the entire internet to work that way. Any user in the world can get any public DNS record for a site.
- remram 4y agoNow if only they could have put the components in the right order...
- throwawaaarrgh 4y agoAs an admin, I use low TTLs when I do some major DNS maintenance. I wouldn't mind upping the TTLs after, but... what about if I need to do some emergency maintenance, like the IPs on a load balancer change and I can't get the old IPs back? I don't want to wait an hour for clients to pick up the new IP. It would be great if there were some reverse notification to let revolvers know to refresh their caches.
- snowwrestler 4y agoWhen I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS. Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before. Also, DNS service is a lot cheaper than it was years ago, so it doesn’t hurt my budget to send more requests back to the name servers.
- account42 4y agoThat sounds like a server setup issue. Nothing about LE prevents you from getting a cert with the old server and moving it to the new server before switchover.
- WirelessGigabit 4y agoUse acme.sh, then all of this happens magically. Also, look at ZeroSSL.
- WirelessGigabit 4y agoHow would a CDN work if TTL is large? Then you run the risk of using a cached IP that isn't close to you.