4 ms·

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had tro

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48. It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice. [0] https://www.ctrl.blog/entry/dns-client-ttl.html https://www.ctrl.blog/entry/dns-client-ttl.html

toast0 4y ago
I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not. If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?
- deleted 4y ago
  [deleted]
- jedberg 4y ago
  That's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.
- donavanm 4y ago
  From the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).
jiggawatts 4y ago
We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like this: > "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem. > "Client software often ignores DNS changes." > "We designed Azure so that Traffic Manager does failover and load balancers are always active-active." > "That's nice, but client software ignores DNS changes." > "Umm..." Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible. The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-region-overview https://learn.microsoft.com/en-us/azure/load-balancer/cross-...
- occamrazor 4y ago
  Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?
  - darkr 4y ago
    It varies; for direct peering/connections where a link goes down (and fast failover protocol is used), usually 5 seconds or so. Global propagation takes in the order of 5 minutes or less: https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-update/ https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-updat...
  - kazen44 4y ago
    routing tables usually aren't "cached" in the same sense that DNS caches entries. however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route. Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).
musicale 4y ago
It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)
charcircuit 4y ago
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
- jedberg 4y ago
  I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
  - Lukas_Skywalker 4y ago
    I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
    - charcircuit 4y ago
      I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.
  - charcircuit 4y ago
    I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out. People without cached records will always get the returned servers with the lowest load. I was referring to load balancing and not splitting traffic for an experiment.
deleted 4y ago
[deleted]
belorn 4y ago
When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them? From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.
hericium 4y ago
> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.
buzzdenver 4y ago
>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.
- account42 4y ago
  Not sure about what kind of connection Alaska has but I imagine this is more about reducing latency for new connections than bandwidth.
jonhohle 4y ago
It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future. 0 - https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html https://docs.oracle.com/javase/7/docs/technotes/guides/net/p...
- pionar 4y ago
  .NET HTTP clients used to suffer from this, too, but they no longer do as of .Net Core 3, as long as they're used correctly.
- nikanj 4y ago
  Only if you have a security manager enabled (=are running in an applet/webstart). The entire security manager concept has been deprecated, so things are looking better
- russellendicott 4y ago
  I seem to recall that HAProxy and Squid used to do this too.
Twirrim 4y ago
The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity. By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control. What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.
throwaway09223 4y ago
> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes. DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design. Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
- mort96 4y ago
  > Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?
  - Serow225 4y ago
    via the magic of Anycast :)
  - throwaway09223 4y ago
    You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example. DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
  - silisili 4y ago
    Yes, with BGP and anycast. What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.
account42 4y ago
> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.
manv1 4y ago
If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box. As the poster says, when it comes to the public internet DNS TTLs are more like guidelines. Luckily most client DNS caches are cleared on a reboot. FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".

toast0 4y ago

I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not. If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?

deleted 4y ago
[deleted]
jedberg 4y ago
That's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.
donavanm 4y ago
From the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).

jiggawatts 4y ago

We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like this: > "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem. > "Client software often ignores DNS changes." > "We designed Azure so that Traffic Manager does failover and load balancers are always active-active." > "That's nice, but client software ignores DNS changes." > "Umm..." Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible. The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-region-overview https://learn.microsoft.com/en-us/azure/load-balancer/cross-...

occamrazor 4y ago
Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?
- darkr 4y ago
  It varies; for direct peering/connections where a link goes down (and fast failover protocol is used), usually 5 seconds or so. Global propagation takes in the order of 5 minutes or less: https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-update/ https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-updat...
- kazen44 4y ago
  routing tables usually aren't "cached" in the same sense that DNS caches entries. however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route. Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).

musicale 4y ago

It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)

charcircuit 4y ago

>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.

jedberg 4y ago
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
- Lukas_Skywalker 4y ago
  I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
  - charcircuit 4y ago
    I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.
- charcircuit 4y ago
  I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out. People without cached records will always get the returned servers with the lowest load. I was referring to load balancing and not splitting traffic for an experiment.

deleted 4y ago

[deleted]

belorn 4y ago

When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them? From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.

hericium 4y ago

> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.

buzzdenver 4y ago

>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.

account42 4y ago
Not sure about what kind of connection Alaska has but I imagine this is more about reducing latency for new connections than bandwidth.

jonhohle 4y ago

It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future. 0 - https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html https://docs.oracle.com/javase/7/docs/technotes/guides/net/p...

pionar 4y ago
.NET HTTP clients used to suffer from this, too, but they no longer do as of .Net Core 3, as long as they're used correctly.
nikanj 4y ago
Only if you have a security manager enabled (=are running in an applet/webstart). The entire security manager concept has been deprecated, so things are looking better
russellendicott 4y ago
I seem to recall that HAProxy and Squid used to do this too.

Twirrim 4y ago

The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity. By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control. What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.

throwaway09223 4y ago

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes. DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design. Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.

mort96 4y ago
> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?
- Serow225 4y ago
  via the magic of Anycast :)
- throwaway09223 4y ago
  You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example. DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
- silisili 4y ago
  Yes, with BGP and anycast. What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.

account42 4y ago

> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.

manv1 4y ago

If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box. As the poster says, when it comes to the public internet DNS TTLs are more like guidelines. Luckily most client DNS caches are cleared on a reboot. FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".