4 ms·

AWS Engineer here. I haven't worked directly on Route 53 in a few years, but I wrote most of the DNS answering logic back when we built Route 53. Route 53 is a

AWS Engineer here. I haven't worked directly on Route 53 in a few years, but I wrote most of the DNS answering logic back when we built Route 53. Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers. I don't think that's in violation of what the DNS RFCs specify (though the older DNS RFCs are not great at being specific), and I remember making the decision based on what would work best, and be least likely to poison resolvers, in the event of domains being moved between customers, providers, or across accounts for some reason. Setting up the child name servers to say "have moved here ..." can speed those processes up. I'm sure the Route 53 team will look at this and make whatever change is best, but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already. In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.

FBISurveillance 9y ago
Thanks for the insights, always interesting to read things like this.
AnkhMorporkian 9y ago
I'm with you. I read the complaint twice, and the RFC twice, and I don't quite think it's a bug. It's a weird thing to reason out, but I don't think Amazon's behavior here is wrong, though it might be unexpected... For various values of unexpected.
agwa 9y ago
> Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers. > I don't think that's in violation of what the DNS RFCs specify Agreed, and in fact it's pretty easy to see that the root and .org name servers behave exactly like Route 53 in this regard. > In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone. Yes, that appears to be what is happening, and it breaks regular DNS resolution, not just QNAME minimization. If your DNS resolver has the NS records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because your resolver will send the queries to the name servers for ws.fdmg.org. I don't know if this is a Route 53 bug or if the customer misconfigured their zones, but if the latter it would be nice if Route 53 could stop people from shooting themselves in the foot this way. To reiterate, this breaks regular DNS resolution, not just QNAME minimization. That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which does break QNAME minimization (and violates RFC8020). For a proper example, see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.
- colmmacc 9y ago
  RFC8020 is an example of the DNS WG making backwards incompatible changes. The "NXDOMAIN" for an empty non-terminal has long been the behavior in DJB's TinyDNS. That isn't what Route 53 is using, but I bring it up because there have been mailing list threads (or flames?) where folks complained about what DJB's software is doing, and he repeatedly asked "Show me where in the standards this is is specified or forbidden?" and no-one would come up with the goods, but folks still basically wanted "Just do what bind does". Here's DJB's own mail about it, I had to forward it because his mails don't get through to the mailing list for some reason (that might be the fault of his own stupid qsecretary moderation system): https://www.ietf.org/mail-archive/web/dnsext/current/msg11084.html https://www.ietf.org/mail-archive/web/dnsext/current/msg1108... I generally sympathize with DJB in this case, because DNS really needs some robust and competing implementations. The standards should be tolerant of differences in behavior (as others are), rather than dominated by the proponents of one implementation. It doesn't help that the WG is toxic, at least in my opinion as someone who has been reading it (and mostly just reading) for over a decade. Anyway, I'm not trying to start an argument, just explain. In this case, the bind proponents did dominate, and we get RFC8020, which for some software is a backwards incompatible change. This isn't the first backwards-incompatible change either. The invention of DNAME introduced a new kind of backwards-incompatible occlusion, and name servers "before" and "after" DNAME respond in different ways, for the same zone file. Those kinds of changes place providers in a pickle: one of our golden rules is never to make backward incompatible changes or break anything that was working. A promise is a promise. So it means we have to lock things down (the previous paragraph is the reason Route 53 does not accept arbitrary literally-encoded RRs) and when changes come; either not make those changes, or exhaustively audit and determine how many zones, queries it might impact and whether that would change anything. At scale, you'd be amazed how small and subtle behaviors can end up being embedded as assumptions in customer scripts and so on.
sigi45 9y ago
Is there any chance that you can add this to your internal "customer want this feature"? I like to be able to add multiply hard/software second factor devices to on IAM Account.
rijswijk 9y ago
> The proper thing a resolver should do is follow the referral and ask the child name servers. You're right, I was clearly not paying attention when writing that part down (fixed now in the forum post). > [...] but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already. AFAIK, that is not the case, QNAME minimisation is well-specified, and the resolver will follow referrals if it gets them. But in this case the resolver is getting an NXDOMAIN where it should be getting either a NODATA response (if there is no zone at ws.fdmg.org), or it should be getting a referral, if there is a zone at ws.fdmg.org, which should then contain a delegation for prod.ws.fdmg.org. Since it's getting neither, the resolver will stop resolution when it gets the NXDOMAIN. Now it may be the case that this is either broken empty non-terminal behaviour, or there is actually a zone at ws.fdmg.org, but that zone is missing a delegation for prod.ws.fdmg.org. It's hard to tell from the outside, but if the latter is the case it should be easy to fix.
- colmmacc 9y ago
  There is a zone cut at ws.fdmg.org, so I think it's the latter. The zone is misconfigured, that's pretty simple to fix for whoever owns the zone, but the general question of whether Route 53 should be more strict and prevent it from being imported or not is a tough call. On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems. In general, at AWS we try to avoid being paternalistic; we believe in giving customers powerful tools and then making those tools intuitive and easy, but I'll admit this can be very challenging with DNS, where the standards and obscure corner cases were never really intended to be understood by an inexpert user-base. I resisted making this point last night because I didn't want to be seen to deflect. But on reflection, the privacy advocate in me really needs to get this out: QNAME minimization is a really really bad idea and doesn't do anything meaningful for privacy IMO. The most privacy sensitive domains are still leaked. DNS queries are still in the clear, including the ultimate specific queries for the domain you want to resolve, observable to anyone who can look at your traffic, maybe including your wifi buddies at the local café. Then along the way the sensitive names are still leaked to parties you probably shouldn't trust in the privacy-protecting model. For example: Verisign can still tell that you are querying for "oddlyspecificpornfetish.com". Dropping the "www", if there even is one, isn't much protection. In return for essentially zero meaningful privacy protection, you have to deal with all sorts of name resolution problems due to making NS queries. That's not a smart trade-off. I'm not saying the goal isn't worthy though, it is.

FBISurveillance 9y ago

Thanks for the insights, always interesting to read things like this.

AnkhMorporkian 9y ago

I'm with you. I read the complaint twice, and the RFC twice, and I don't quite think it's a bug. It's a weird thing to reason out, but I don't think Amazon's behavior here is wrong, though it might be unexpected... For various values of unexpected.

agwa 9y ago

> Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers. > I don't think that's in violation of what the DNS RFCs specify Agreed, and in fact it's pretty easy to see that the root and .org name servers behave exactly like Route 53 in this regard. > In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone. Yes, that appears to be what is happening, and it breaks regular DNS resolution, not just QNAME minimization. If your DNS resolver has the NS records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because your resolver will send the queries to the name servers for ws.fdmg.org. I don't know if this is a Route 53 bug or if the customer misconfigured their zones, but if the latter it would be nice if Route 53 could stop people from shooting themselves in the foot this way. To reiterate, this breaks regular DNS resolution, not just QNAME minimization. That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which does break QNAME minimization (and violates RFC8020). For a proper example, see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.

colmmacc 9y ago
RFC8020 is an example of the DNS WG making backwards incompatible changes. The "NXDOMAIN" for an empty non-terminal has long been the behavior in DJB's TinyDNS. That isn't what Route 53 is using, but I bring it up because there have been mailing list threads (or flames?) where folks complained about what DJB's software is doing, and he repeatedly asked "Show me where in the standards this is is specified or forbidden?" and no-one would come up with the goods, but folks still basically wanted "Just do what bind does". Here's DJB's own mail about it, I had to forward it because his mails don't get through to the mailing list for some reason (that might be the fault of his own stupid qsecretary moderation system): https://www.ietf.org/mail-archive/web/dnsext/current/msg11084.html https://www.ietf.org/mail-archive/web/dnsext/current/msg1108... I generally sympathize with DJB in this case, because DNS really needs some robust and competing implementations. The standards should be tolerant of differences in behavior (as others are), rather than dominated by the proponents of one implementation. It doesn't help that the WG is toxic, at least in my opinion as someone who has been reading it (and mostly just reading) for over a decade. Anyway, I'm not trying to start an argument, just explain. In this case, the bind proponents did dominate, and we get RFC8020, which for some software is a backwards incompatible change. This isn't the first backwards-incompatible change either. The invention of DNAME introduced a new kind of backwards-incompatible occlusion, and name servers "before" and "after" DNAME respond in different ways, for the same zone file. Those kinds of changes place providers in a pickle: one of our golden rules is never to make backward incompatible changes or break anything that was working. A promise is a promise. So it means we have to lock things down (the previous paragraph is the reason Route 53 does not accept arbitrary literally-encoded RRs) and when changes come; either not make those changes, or exhaustively audit and determine how many zones, queries it might impact and whether that would change anything. At scale, you'd be amazed how small and subtle behaviors can end up being embedded as assumptions in customer scripts and so on.

sigi45 9y ago

Is there any chance that you can add this to your internal "customer want this feature"? I like to be able to add multiply hard/software second factor devices to on IAM Account.

rijswijk 9y ago

> The proper thing a resolver should do is follow the referral and ask the child name servers. You're right, I was clearly not paying attention when writing that part down (fixed now in the forum post). > [...] but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already. AFAIK, that is not the case, QNAME minimisation is well-specified, and the resolver will follow referrals if it gets them. But in this case the resolver is getting an NXDOMAIN where it should be getting either a NODATA response (if there is no zone at ws.fdmg.org), or it should be getting a referral, if there is a zone at ws.fdmg.org, which should then contain a delegation for prod.ws.fdmg.org. Since it's getting neither, the resolver will stop resolution when it gets the NXDOMAIN. Now it may be the case that this is either broken empty non-terminal behaviour, or there is actually a zone at ws.fdmg.org, but that zone is missing a delegation for prod.ws.fdmg.org. It's hard to tell from the outside, but if the latter is the case it should be easy to fix.

colmmacc 9y ago
There is a zone cut at ws.fdmg.org, so I think it's the latter. The zone is misconfigured, that's pretty simple to fix for whoever owns the zone, but the general question of whether Route 53 should be more strict and prevent it from being imported or not is a tough call. On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems. In general, at AWS we try to avoid being paternalistic; we believe in giving customers powerful tools and then making those tools intuitive and easy, but I'll admit this can be very challenging with DNS, where the standards and obscure corner cases were never really intended to be understood by an inexpert user-base. I resisted making this point last night because I didn't want to be seen to deflect. But on reflection, the privacy advocate in me really needs to get this out: QNAME minimization is a really really bad idea and doesn't do anything meaningful for privacy IMO. The most privacy sensitive domains are still leaked. DNS queries are still in the clear, including the ultimate specific queries for the domain you want to resolve, observable to anyone who can look at your traffic, maybe including your wifi buddies at the local café. Then along the way the sensitive names are still leaked to parties you probably shouldn't trust in the privacy-protecting model. For example: Verisign can still tell that you are querying for "oddlyspecificpornfetish.com". Dropping the "www", if there even is one, isn't much protection. In return for essentially zero meaningful privacy protection, you have to deal with all sorts of name resolution problems due to making NS queries. That's not a smart trade-off. I'm not saying the goal isn't worthy though, it is.