DNS seems ripe for revolution! Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am not intimately familiar with the implementations

DNS seems ripe for revolution! Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am not intimately familiar with the implementations in BIND and others, but it seems like when we hit the TTL we just throw away the data. Usually, that works fine. But, in the case of the origin servers not responding, yesterday I was wishing that it would just give me back the stale data rather than giving me nothing. I'll admit the impact on me was somewhat limited. Around 10am Mountain I was trying to install some Atom.io modules and couldn't reach that site or a github download URL. I had some success with using 4.2.2.2 (8.8.8.8 was not answering the names). Using a stale cached result probably wouldn't have helped for atom.io though, I hadn't been there in a while and this was querying my own local name servers. Do I want my name servers keeping weeks old stale data around? Probably not in RAM, but saving old names to disk sounds like it would require a lot of IOPS for a big provider. But I do know I'd visited the sites I was trying to hit within the last few weeks, since I couldn't reach the authoritative servers it'd be nice to have tried the last IP I had for them. Of course, I ran into this about an hour after I rebooted my entire dev/staging infrastructure to fix the Linux kernel privilege escalation issue, so my caches were cold. Sure would be nice if my server could "ask around" if it can't talk to an authoritative server. "Hey Google, hey Comcast, hey Level-3, do you know this name?" That's effectively what I did by changing my resolv.conf. But if you start asking around too widely, you ideally probably want to have some signature to verify the data you are getting. Seems like the new norm might be listing authoritative DNS servers from multiple big providers (Dyn and Route53) and having to keep them in sync? Then you lose some of the advance features... Funny aside: One of the sites I run uses Distil in front of it to protect against content scrapers. Months ago I was working with their support about getting a health checker set up in Route53 to test the full paper path through distil and fail over to our backup site if anything on the primary paper path didn't work. Distil assured me that their services were so resilient that we shouldn't worry about them being down. Guess what the only part of our infrastructure was that was impacted by this? :-)