6 ms·
AWS DNS services break qname minimization
- EwanToo 9y agoI had honestly never heard of the term qname minimisation before, but this blog post gives a pretty clear explanation of the benefits and issues https://blog.verisign.com/security/minimum-disclosure-what-information-does-a-name-server-need-to-do-its-job/ https://blog.verisign.com/security/minimum-disclosure-what-i...
- jlgaddis 9y agoSee also ("experimental") RFC7816 [0] for all the gory details. [0]: https://tools.ietf.org/html/rfc7816 https://tools.ietf.org/html/rfc7816
- merb 9y agowell I'm not sure if it's sane to implement this rfc. it's experimental since march 2016. and I do not think that privacy should be a concern when resolving... I mean what is private about the dns system itself?!
- takeda 9y agoJust to clarify, the qname minimization is a privacy technique used during recursion to only disclose what's absolutely necessary. Traditionally if you're querying www.google.com, you'll: 1. ask root name servers for www.google.com and get list of com name servers 2. ask com name servers for www.google.com and get list of google name servers 3. ask google name servers for www.google.com and get the response With qname minimization you would: 1. ask root name servers for com TLD and get list of com name servers 2. ask com name servers for google.com and get list of google name servers 3. ask google name servers for www.google.com and get the response The AWS DNS servers don't strictly follow rfc and don't provide correct responses in specific cases
- feelin_googley 9y agoTry counting number of DNS queries for all lookups every day. The software I wrote for myself does this and I can attest that sites using AWS nameservers in almost all cases require more queries than others, including those using similar managed DNS services. The number, which does not include queries to the root can be quite high, e.g., eight. No matter how fast the response times, that delay adds up. root.zone data is reasonably static. Having a local copy of this data, resolution can be done in two queries: 1. tld ns 2. auth ns, which is the case for many, many www domainnames (the majority, perhaps), or even one query (I think some .mil domains used to be a well-known example). As a user, I view every added query as an additional cost. It is possible to minimize queries via other means that are under user control and do not rely on third parties. Running a root locally. Custom stub resolvers with tld ns IPs compiled in. Storing RRs for common nameservers in local zone. Storing DNS data in bulk for important sites that will be revisited, copying to HOSTS and updating the data periodically. I have been using techniques like these for many years. And, it is getting easier every year as more bulk DNS data is available, e.g., internet scans of DNS, TCP DNS services that accept pipelined queries, etc. (If I am not mistaken, the author of the "qname minimisation" draft also was the one with the "DNS looking glass" idea.) https://tools.ietf.org/html/draft-ietf-dnsop-qname-minimisation https://tools.ietf.org/html/draft-ietf-dnsop-qname-minimisat... Despite usual mailing list/forum comment/blog memes and mantras, if one surveys all websites (not just popular ones), DNS data is mostly static. That provides great opportunities for speedups by avoiding querying ICANN DNS for the same data 24/7/365. Cue Schopenhauer quote about truth. Someday maybe we will see an internet draft about "DNS minimization" for users. Did AWS forums start requiring an account for read access?
- becauseiam 9y ago> Did AWS forums start requiring an account for read access? No but if you timed out from a session with your account through the console, you might see something about not being signed in. Clicking the "log out" button (or hosing cookies) should rectify it.
- deleted 9y ago[deleted]
- FiloSottile 9y agoHeh, this is a fairly easy mistake to make when implementing a DNS server. I’m the person that had to fix this very same bug (and possibly introduced it) for a certain class of domains in the Cloudflare DNS server. “Empty non-terminal” still has an emotional reaction attached to it. :) Also, pretending all NXDOMAINs are actually empty non-terminals is a backwards compatible hack to compress DNSSEC responses https://tools.ietf.org/html/draft-valsorda-dnsop-black-lies-00 https://tools.ietf.org/html/draft-valsorda-dnsop-black-lies-...
- ttul 9y agoAWS DNS is all kinds of fucked. They have implemented a range of undocumented rate limits both to their own DNS server and to external (!) DNS, presumably to reduce amplification attacks. These things have hobbled our services many times, which rely on making thousands of DNS queries per second.
- boulos 9y agoCan you describe a bit more about what you’re resolving? And by rate-limiting external, are you saying they’re throttling DNS traffic to say 8.8.8.8 as well, or do you mean external as opposed to “internal” in a split horizon sense?
- ttul 9y agoThis is correct.
- otterley 9y agoI'm curious about your architecture. What requires you to make thousands of DNS queries per second against Route 53?
- colmmacc 9y agoAWS Engineer here. I haven't worked directly on Route 53 in a few years, but I wrote most of the DNS answering logic back when we built Route 53. Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers. I don't think that's in violation of what the DNS RFCs specify (though the older DNS RFCs are not great at being specific), and I remember making the decision based on what would work best, and be least likely to poison resolvers, in the event of domains being moved between customers, providers, or across accounts for some reason. Setting up the child name servers to say "have moved here ..." can speed those processes up. I'm sure the Route 53 team will look at this and make whatever change is best, but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already. In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.
- FBISurveillance 9y agoThanks for the insights, always interesting to read things like this.
- AnkhMorporkian 9y agoI'm with you. I read the complaint twice, and the RFC twice, and I don't quite think it's a bug. It's a weird thing to reason out, but I don't think Amazon's behavior here is wrong, though it might be unexpected... For various values of unexpected.
- agwa 9y ago> Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers. > I don't think that's in violation of what the DNS RFCs specify Agreed, and in fact it's pretty easy to see that the root and .org name servers behave exactly like Route 53 in this regard. > In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone. Yes, that appears to be what is happening, and it breaks regular DNS resolution, not just QNAME minimization. If your DNS resolver has the NS records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because your resolver will send the queries to the name servers for ws.fdmg.org. I don't know if this is a Route 53 bug or if the customer misconfigured their zones, but if the latter it would be nice if Route 53 could stop people from shooting themselves in the foot this way. To reiterate, this breaks regular DNS resolution, not just QNAME minimization. That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which does break QNAME minimization (and violates RFC8020). For a proper example, see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.
- nvarsj 9y agoTangentially related, DNS quirks show up in lots of places. For example, FB’s graph API doesn’t work with FQDN: * https://graph.facebook.com./v2.8 - returns 500 * https://graph.facebook.com/v2.8 - works (I filed a bug for this, but it was closed as “working as designed”).