4 ms·

Same, saw a DNS entry for an internal DB endpoint get updated with an 8 hour TTL for a planned failover. DB Admins went to sleep as everything was fine, everyon

by jabart 6y ago

Same, saw a DNS entry for an internal DB endpoint get updated with an 8 hour TTL for a planned failover. DB Admins went to sleep as everything was fine, everyone woke up 8 hours later with everything failing to connect. Had to flush the DNS on each internal server to fix that issue.

cookiengineer 6y ago
The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.
- rhacker 6y ago
  I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly static forever. We don't live in the old world where we had to take a server down any more. It's all managed.
  - laurent92 6y ago
    What about 2 minutes during the first hour after a change, 30 minutes during 2 days, 3 hrs after that?
    - squiggleblaz 6y ago
      I think the point is that I don't want to change my TTLs. If I had my druthers, yeah, they'd go up periodically, and then if tech support receives an issue that sounds like it might require DNS changes, they press a button and the TTL drops. Once that issue is closed, it goes back to gradually increasing.
    - toast0 6y ago
      This would be fine for some uses. It's not great for client facing names where you want to be able to react to an incident and have traffic move over quickly. If it's going to take more than 3 hours to setup the new traffic target anyway, sure. You have time to fiddle with TTLs. If you have servers ready to go elsewhere, 1-5 minute TTLs are nice so you can quickly move things when you notice a problem.
  - jabart 6y ago
    During the Centurylink outage, like last year?, I had to switch IPs to a different network since those were exposed over BGP and you couldn't update BGP anymore. All one offs sure, the point of low TTLs is to be prepared. I pay for Route53 by the query so I'm aware of the cost and it's still nothing compared to the cost of an outage waiting for a TTL to expire.
  - cogman10 6y ago
    I just don't see the harm in a short TTL. Most apps are "bursty" so a 1 minute TTL is more than ample, basically giving them a 1 time penalty on the first request then nothing for the rest of the requests. 1 minute is a long time, you can do a ton of requests in 60,000 milliseconds. On the flip side, setting the TTL long can be a disaster. You can't fix it after the fact. If you have a 1 hour TTL then that's potentially 1 hour before the changes needed to fix a service fully take effect. That's 1 hour of helplessness.
    - mlyle 6y ago
      Surely we can live with 5-10 minutes for most things, though, right?
      - cogman10 6y ago
        I don't see why not. I just don't see there being a huge difference between 1 and 10 minutes in the grand scheme of tech.
        pixl97 6y ago
        That's an order of magnitude which can make a pretty big difference when demand spikes.
    - mjevans 6y ago
      It would be nice if failure to connect / TLS handshake failure invalidated the entire related network stack cache for that link. If that were the case I wouldn't mind higher TTLs.
- sparrish 6y ago
  Migration accidents happen. If your TTLs are long, those accidents cause long disruptions and big downtimes. That's a poor customer experience. I definitely think it's worth the millions or billions of DNS requests. Use short TTLs, your customers will thank you.
- xoa 6y ago
  I'd kind of like if there was some ability to have more complex multi-part TTLs as an option along with a default TTL (the current one). So I could specify Default TTL: 12 hours; [<startdate> to <enddate>] TTL: 10 minutes; or even Default TTL: 12 hours; [Thu 0000-1200, repeating]: 5 minutes; [<startdate> to <enddate>] TTL: 10 minutes; So with no further effort all downstream caches/clients can basically have advanced notice of regular maintenance windows as well as planned maintenance and just automatically adapt. Of course this wouldn't deal with true emergencies, but it might lower the overhead for a huge amount of regular stuff that otherwise tempts people to set it low and leave it that way. I dunno, I'm sure there's other downsides I haven't thought of, and proper implementation would require thinking through side effects. But after a long time dealing with it feels like there's some room for something beyond one single TTL ever which must be specifically changed (with a wait for propagation) well ahead of time whenever anything planned with a risk of issues needs to be done. Maybe?
  - Diggsey 6y ago
    I don't think you'd need to further complicate DNS, just have a service running that checks a calendar and syncs your DNS TTLs appropriately. eg. - Check if there's a maintenance window in the next [max TTL] - If not, set TTL = [max TTL] - Otherwise, set TTL = [time until maintenance window] - Repeat
  - deleted 6y ago
    [deleted]

by jabart 6y ago

cookiengineer 6y ago

The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.

rhacker 6y ago
I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly static forever. We don't live in the old world where we had to take a server down any more. It's all managed.
- laurent92 6y ago
  What about 2 minutes during the first hour after a change, 30 minutes during 2 days, 3 hrs after that?
  - squiggleblaz 6y ago
    I think the point is that I don't want to change my TTLs. If I had my druthers, yeah, they'd go up periodically, and then if tech support receives an issue that sounds like it might require DNS changes, they press a button and the TTL drops. Once that issue is closed, it goes back to gradually increasing.
  - toast0 6y ago
    This would be fine for some uses. It's not great for client facing names where you want to be able to react to an incident and have traffic move over quickly. If it's going to take more than 3 hours to setup the new traffic target anyway, sure. You have time to fiddle with TTLs. If you have servers ready to go elsewhere, 1-5 minute TTLs are nice so you can quickly move things when you notice a problem.
- jabart 6y ago
  During the Centurylink outage, like last year?, I had to switch IPs to a different network since those were exposed over BGP and you couldn't update BGP anymore. All one offs sure, the point of low TTLs is to be prepared. I pay for Route53 by the query so I'm aware of the cost and it's still nothing compared to the cost of an outage waiting for a TTL to expire.
- cogman10 6y ago
  I just don't see the harm in a short TTL. Most apps are "bursty" so a 1 minute TTL is more than ample, basically giving them a 1 time penalty on the first request then nothing for the rest of the requests. 1 minute is a long time, you can do a ton of requests in 60,000 milliseconds. On the flip side, setting the TTL long can be a disaster. You can't fix it after the fact. If you have a 1 hour TTL then that's potentially 1 hour before the changes needed to fix a service fully take effect. That's 1 hour of helplessness.
  - mlyle 6y ago
    Surely we can live with 5-10 minutes for most things, though, right?
    - cogman10 6y ago
      I don't see why not. I just don't see there being a huge difference between 1 and 10 minutes in the grand scheme of tech.
      - pixl97 6y ago
        That's an order of magnitude which can make a pretty big difference when demand spikes.
  - mjevans 6y ago
    It would be nice if failure to connect / TLS handshake failure invalidated the entire related network stack cache for that link. If that were the case I wouldn't mind higher TTLs.
sparrish 6y ago
Migration accidents happen. If your TTLs are long, those accidents cause long disruptions and big downtimes. That's a poor customer experience. I definitely think it's worth the millions or billions of DNS requests. Use short TTLs, your customers will thank you.
xoa 6y ago
I'd kind of like if there was some ability to have more complex multi-part TTLs as an option along with a default TTL (the current one). So I could specify Default TTL: 12 hours; [<startdate> to <enddate>] TTL: 10 minutes; or even Default TTL: 12 hours; [Thu 0000-1200, repeating]: 5 minutes; [<startdate> to <enddate>] TTL: 10 minutes; So with no further effort all downstream caches/clients can basically have advanced notice of regular maintenance windows as well as planned maintenance and just automatically adapt. Of course this wouldn't deal with true emergencies, but it might lower the overhead for a huge amount of regular stuff that otherwise tempts people to set it low and leave it that way. I dunno, I'm sure there's other downsides I haven't thought of, and proper implementation would require thinking through side effects. But after a long time dealing with it feels like there's some room for something beyond one single TTL ever which must be specifically changed (with a wait for propagation) well ahead of time whenever anything planned with a risk of issues needs to be done. Maybe?
- Diggsey 6y ago
  I don't think you'd need to further complicate DNS, just have a service running that checks a calendar and syncs your DNS TTLs appropriately. eg. - Check if there's a maintenance window in the next [max TTL] - If not, set TTL = [max TTL] - Otherwise, set TTL = [time until maintenance window] - Repeat
- deleted 6y ago
  [deleted]