15 ms·
Out of curiosity, why do caching DNS resolvers, such as the DNS resolver I run on my home network, not provide an option to retain last-known-good resolutions b
by bhauer 10y ago
Out of curiosity, why do caching DNS resolvers, such as the DNS resolver I run on my home network, not provide an option to retain last-known-good resolutions beyond the authority-provided time to live? In such a configuration, after the TTL expiration, the resolver would attempt to refresh from the authority/upstream provider, but if that attempt fails, the response would be a more graceful failure of returning a last-known-good resolution (perhaps with a flag). This behavior would continue until an administrator-specified and potentially quite generous maximum TTL expires, after which nodes would finally see resolution failing outright.
Ideally, then, the local resolvers of the nodes and/or the UIs of applications could detect the last-known-good flag on resolution and present a UI to users ("DNS authority for this domain is unresponsive; you are visiting a last-known-good IP provided by a resolution from 8 hours ago."). But that would be a nicety, and not strictly necessary.
Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside.
- drinchev 10y agoAll will break the moment that one of the websites that you access makes a server-side request to another website ( think about logging-services, server-clusters, database servers, etc - they all either have IPs or most-likely some domains. )
- bhauer 10y agoI'm not sure I understand what you're saying. The scenario is that my local network's caching DNS resolver retains resolutions beyond the authority-provided TTL in the event that a TTL-specified refresh at expiration fails. Therefore, my web browser may—in the very rare situation where this arises—make an HTTP request to an IP address of a server that has been intentionally moved by a service provider (let's assume they did so expecting their authoritative TTL to have expired). Since this scenario only arises because my caching resolver wasn't able to reach the authority, I'm not seeing a downside. But if I understand your reply correctly, you are saying that the web server I've contacted may, in turn, be using a DNS resolver that is similarly configured to provide last-known-good resolution when its upstream provider/authority cannot provide resolution. This would potentially result in that web server making an HTTPS API request to a wrong IP, again only in the rare case where we have defaulted to a last-known-good resolution. I'm not really seeing the problem here except that the HTTPS request might fail (if the expected service was moved and no longer exists at the last-known-good IP), but how is that worse than the DNS resolution having failed? In both cases, the back-end service request fails.
- leesalminen 10y agoOne possible issue is that IPs are re-used in cloud environments. Potentially, your browser could POST sensitive data to an IP address that now belongs to a totally different company.
- mej10 10y agoYeah, that is definitely possible. I mean, hopefully it is over HTTPS so they can't do anything with it... but if it isn't then it can definitely happen. Our servers get random web traffic all of the time.
- bhauer 10y agoTrue. I am not aware of any web services POSTing sensitive data over the public Internet that don't use HTTPS. If your service is sending sensitive data over HTTP without TLS, I feel the problem is bigger than a potential long-lived DNS resolution.
- leesalminen 10y agoValid point! One would hope that does not happen.
- woot01 10y agoHTTPS does not protect you against sending data to a host owned by another company.
- leesalminen 10y agoCorrect, but they wouldn't be able to decrypt the data.
- icebraining 10y agoThe data doesn't even get there, the handshake kills the connection before that.
- beachstartup 10y agoi seem to remember that dns has generally been reliable (until recently, i guess), probably nobody has ever thought that to be necessary. you could write a cron script that generates a date-stamped hosts file based on a list of your top-used domain names, and simply use that on your machine(s) if your dns ever goes down. that's basically a very simple local dns cache. if you feel like living dangerously, have it update /etc/hosts directly.
- caleblloyd 10y ago> i seem to remember that dns has generally been reliable (until recently, i guess) Probably because people used to use long TTLs (1 hour, 4 hours, whatever) and now the default behavior in services like Amazon Route 53 is to use 5 minutes.
- chetanahuja 10y agoTry Akamai managed CDN content. 20 seconds !!
- Edmond 10y agoThe 20 seconds with Akamai is because of their dynamic end user IP mapping technology, Basically they need to map in near real-time based on characteristics of the end user IP, they can't afford a long TTL
- chetanahuja 10y agoIt's not illegal to have TTL that short but it certainly feels like violation of some implicit contract between users and provides. Of course the root cause of this is the horrendous hack of using DNS for CDN routing. It doesn't have to be that way... I wrote a recent article about this very issue here http://www.infoworld.com/article/3133104/mobile-technology/why-your-mobile-connection-is-so-slow.html http://www.infoworld.com/article/3133104/mobile-technology/w...
- the_mitsuhiko 10y ago> Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside. Because you would keep old DNS records around forever if a server goes away for good. So you need to have a timeout for that anyways.
- bhauer 10y agoYes, but: 1) Memory and disk are cheap. My caching DNS resolver can handle some stale records. 2) I suggested above that this behavior would continue until an administrator-specified and potentially quite generous maximum TTL expires. That is, I could configure my caching DNS resolver to fully purge expired records after, for example, 2 weeks.
- the_mitsuhiko 10y ago> 1) Memory and disk are cheap. My caching DNS resolver can handle some stale records. The problem is not that it would require storage but that stale records can be outright wrong. That timeout would require configuration and DNS does not provide that. So sure, a new timeout could be introduced but that currently does not exist in DNS.
- joncrane 10y agoI think what the poster above you is saying is a feature on some software that isn't in an RFC somewhere.
- bhauer 10y ago> The problem is not that it would require storage but that stale records can be outright wrong. Again, the scenario is that the authoritative/upstream resolver cannot be reached in order to refresh after the authority-provided TTL expires. Are you saying that in the case of a service having been intentionally removed from the Internet (the domain is deactivated; the service is simply no more), my caching resolver will continue to resolve the domain for a time? Yes, it would. What's the downside though? > That timeout would require configuration and DNS does not provide that. Yes. This would be a configurable option in my caching DNS resolver, in the same vein as specifying the forwarders, roots, and so on. But to be clear, this would not be a change to the DNS protocol, merely a configuration change to control the cache expiration behavior of my resolver. I'm not wanting to sound flippant, I'm not sure I understand the point you're trying to make here.
- Kalium 10y agoHistorically, doing this has been a source of a truly awe-inspiring amount of pain.
- gmaslov 10y agoAw, don't leave us hanging like that. What problems did it cause?
- Kalium 10y agoImagine migrating your website to a new host. A month later, you learn that a major ISP has decided that its customers don't need to know about the move, because they hold on to last-known-good records as they like. So half your traffic and business is gone. Or maybe you can't use anything run on Heroku, because the dynamicism there doesn't play nice with your resolver's policies. That's the kind of world we used to live in when TTLs were often treated as vague suggestions.
- bhauer 10y agoThe scenario I was describing was one where a last-known-good resolution would be used if and only if a refresh attempt fails after the authority-provided TTL expires. I believe the scenario you are describing is a rogue ISP ignoring that authoritative TTL wholesale, caching resolutions according to its own preferences regardless of whether the authority is able to provide a response after the authoritative TTL expires.
- Kalium 10y agoThe rogue ISPs thought they were helping people by serving stale data. After all, better something past its use-by date than failing, right? A low tolerance for DNS response times, and suddenly large chunks of the internet are failing a lot... Among other problems, this enables attacks. Leak a route, DDoS a DNS provider, and watch as traffic everywhere goes to an attack server because servers everywhere "protect" people by serving known-stale data rather than failing safe. Be very, very careful when trying to be "safer". It can unintentionally lead somewhere very different.
- davidu 10y agoOpenDNS does this: https://support.opendns.com/hc/en-us/articles/227987767-Dynamic-IP-Addresses-Technical-Detail-and-FAQ https://support.opendns.com/hc/en-us/articles/227987767-Dyna... It's called SmartCache.
- bhauer 10y agoAwesome! Is this available as software I can install on my network? Sorry, probably a dumb question.
- imglorp 10y agoNope, just point your machine or router's DNS to use opendns resolvers instead of your regular ones: 208.67.222.222 and 208.67.220.220
- deleted 10y ago[deleted]
- pitosalas 10y agoDo you have a link on the opendns web site that refers to those specific Ips?
- jasonellis 10y agohttps://www.opendns.com/setupguide/ https://www.opendns.com/setupguide/
- efoto 10y agoOne can go even further and install DNSCrypt: https://dnscrypt.org/ https://dnscrypt.org/
- saganus 10y agoAny downsides to using this? I'm tempted to start using it, but I'm not really sure if there's any particular thing I should consider first.
- bluejekyll 10y agoI've been thinking of adding this exact feature to my DNS framework that I've been working on (if github was resolving): https://github.com/bluejekyll/trust-dns https://github.com/bluejekyll/trust-dns If you have any feedback, I'd love to hear it.
- bhauer 10y agoSounds promising. I'll want to take a look when GitHub is resolvable again. :)
- Kalium 10y agoTo be perfectly honest, a "feature" like this has no business being in a safe and secure DNS server. You should fail-safe, rather than serving stale data of unknown safety. Serving data you cannot verify is a dangerous failure state.
- bluejekyll 10y agoPerhaps. In this case the web was down. I definitely understand the point, stale data with TTLs which have expired, especially on RRSIG records is dangerous. But I have to wonder about situations like this where DNS has been taken down, what the better good is. If the records can be proven to have been cached as authentic data at some point within some period of time. In this case hours, is it for the better good that stale authentic records are acceptable to serve back? In this case a stale period of some number of hours would have been good. I'm not so sure which is better in this case.
- Kalium 10y agoWould you believe me if I said that the DNS protocol itself has an answer for this? The answer is in the basic design of what a TTL is. It's preferable to serve nothing than to serve known bad data. Stale data is a form of bad data. As another user put it, we have these standards for good reason.
- zwily 10y agoThis can be pretty bad in a world where AWS ELB IP addresses change regularly.
- cjbprime 10y agoWhy? The OP is only proposing using a cached result when there's no updated record available.
- toomuchtodo 10y agoServing wrong records is usually worse than serving no records. EDIT: It would be fine as long as your site only served HTTPS content and HSTS was enabled for your domain, preventing any sort of MITM attack.
- cjbprime 10y agoIf your DNS server is offline, is the last record it returned when it was online really the "wrong" one? There'd be no right one in that case.
- detaro 10y agoExactly. You can't know if it is still valid, so you might send clients to an IP that's now controlled by somebody else. Worst case, they know and set up a phishing site. DNS generally has been reliable enough that the trade-off is not worth it.
- mynameisvlad 10y ago> Worst case, they know and set up a phishing site. They'd need to specifically gain access to the last known good IP address, which might be different depending on which DNS resolver you talk to (geodistribution, when the record was last updated, etc). I wouldn't really consider that a realistic attack vector.
- jasimp 10y agoI can guarantee you that popular DNS resolvers (think 500b+ transactions a day) do have this feature! Don't want to say much more due to it being my job, and I don't want to give away too much. EDIT: https://www.google.com/patents/US8583801 https://www.google.com/patents/US8583801
- imsofuture 10y agoWhat is the point of this comment?
- bhauer 10y agoFWIW, since the comment was a reply to my message above: It provided value by answering my question concerning serious downsides to providing optional post-TTL last-known-good caching within a DNS resolver. The answer is implicit in that a major DNS resolver provides exactly this functionality.
- jasimp 10y agoThank you :) A little more information, considering it is public. (I had to double check if it was) https://www.google.com/patents/US8583801 https://www.google.com/patents/US8583801
- pyvpx 10y agoyou mean opendns?
- jasimp 10y agoNo I mean ISP's that run their own DNS resolvers.
- JoshTriplett 10y agoIt'd be nice to have a "backup TTL" included, to allow sites to specify whether and how long they wanted such caching behavior. Also, that cache would need to only kick in when the server was unreachable or produced SERVFAIL, not when it returned a negative result. Negative results returned by the authoritative server are correct, and should not result in the recursive resolver returning anything other than a negative result.
- bhauer 10y ago> Also, that cache would need to only kick in when the server was unreachable or produced SERVFAIL, not when it returned a negative result. Negative results returned by the authoritative server are correct, and should not result in the recursive resolver returning anything other than a negative result. Precisely. I am not suggesting any change to how a caching resolver comprehends valid responses from the authoritative server for a domain. For example, if the authoritative server says, "No such domain," then the domain is understood to be gone. At that point, the domain being gone is in fact the last-known-good resolution.
- spc476 10y agoIt might be a stretch to use the information, but the SOA RR does contain an EXPIRE field, defined as "A 32 bit time value that specifies the upper limit on the time interval that can elapse before the zone is no longer authoritative." It's an additional request, but the SOA RR does contain the type of information you are asking for.
- Godel_unicode 10y agoThis is why TTL is a variable. If you want to have your records last longer, you can. If you want to later shorten them, you can. People screw DNS up enough already, let's not make it worse by adding layers of TTL.
- LordHumungous 10y agoI think a problem that you might be overlooking is that DNS lookups aren't just failing, they are also very slow when a DDOS attack is underway on the authority servers. This introduces a latency shock to the system which causes cascading failures.
- DivineTraube 10y agoHTTP has a good solution/proposal for this: the server can include a stale-on-error=someTimeInSeconds header in addition to the TTL and then every cache is allowed to continue serving stale data for the specified time while the origin is unreachable. Probably a good idea to include such a mechanism in DNS, too. https://tools.ietf.org/html/rfc5861 https://tools.ietf.org/html/rfc5861
- jedisct1 10y agoYou can install EdgeDNS locally. It does that, among other things. https://github.com/jedisct1/edgedns https://github.com/jedisct1/edgedns