6 ms·

These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver

by wrs 9y ago

These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. But their misconfiguration happens to work most of the time for their use case. The solution is either to implement a "bug-compatible" priority flag in resolved, or to make that a first-class use case with its own flag ("emergency backup nameserver"?). It's a classic case...should the software follow the spec, or the real-world usage? And people complain either way.

kelnos 9y ago
Real-world usage, obviously. If you're going to replace a foundational component of the OS, you need to replace it in a bug-compatible way, period. No compromises. A common use for this behavior is VPNs: you sign onto a VPN, and the VPN's DNS server gets prepended to the list in resolv.conf, which allows you to resolve internal addresses (and still fall back to your regular DNS). This is a crazy-common use case, and it's incredibly arrogant of Poettering to just dismiss it out of hand, or even attempt to argue with it.
- tssva 9y ago
  Which real world usage? dnsmasq and Windows both follow the systemd behavior by default.
- wrs 9y ago
  Our VPN is configured to replace the server list, which is the correct behavior, because the new servers don't return the same answers as the old ones. So again, those other VPNs are misconfigured.
- geofft 9y ago
  On macOS, there's a straightforward solution for the VPN use case that doesn't involve a "fall back": the VPN registers its DNS servers for specific domain names, and so anything in example.com has its list of DNS servers (tunneled through the VPN) and everything else has the original list of DNS servers from your physical network connection. See resolver(5) https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man5/resolver.5.html https://developer.apple.com/legacy/library/documentation/Dar... for a description of doing this with files named like "/etc/resolver/example.com" (and having the same syntax as resolv.conf), but you can also do this with SystemConfiguration (googling "SupplementalMatchDomains" should be helpful). It looks like systemd-resolved implements something similar: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net... The fact that there is an incredibly common use case that consists of an unreliable workaround for people who don't have a local resolver that does this correctly is hardly reason to avoid writing a local resolver that works correctly.
  - kelnos 9y ago
    If you have an incredibly simple internal network, that might work fine. It certainly would not work at my organization.
subway 9y ago
No, this is not a misconfiguration. It is a configuration that expects the resolver to behave according to the algorithm described in resolver(5). systemd chose to wedge itself in as a resolver, then broke that contract.
- jerf 9y ago
  I was curious about this, since it sort of impacts my wourld, and decided to check this. There's a RES_ROTATE flag in there, described as "Causes round-robin selection of name servers from among those listed. This has the effect of spreading the query load among all listed servers, rather than having all clients try the first listed server first every time." Claim checks out.
- wrs 9y ago
  resolver(5) describes the behavior of glibc, not DNS or systemd. Systemd will read resolv.conf if you don't want to configure resolved.conf directly, but that's a convenience, not a contract. Dnsmasq also reads resolv.conf for convenience, and puts yet another interpretation on it (which will also break this use case). Both systemd and dnsmasq expect that you'll tell glibc to just ask the local resolver (them).
joombaga 9y ago
> all DNS servers are expected to return the same result Really? My local DNS server should return the same results as 8.8.8.8? Should one never use unregistered suffixes?
- JdeBP 9y ago
  All listed proxy DNS servers should provide the same view of the DNS namespace. If your local proxy DNS server provides a different view to Google Public DNS, then it is wrong to list them together. * http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-same-service.html http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-sam... And no, you shouldn't just blithely assume that you own parts of the DNS namespace that you do not, either. There are plenty of other people's mistakes to learn from here. Don't repeat them. * http://jdebp.eu./FGA/dns-use-domain-names-that-you-own.html http://jdebp.eu./FGA/dns-use-domain-names-that-you-own.html
- rosser 9y ago
  I think the argument is that if you do put both your local DNS server and 8.8.8.8 in your /etc/resolv.conf, then yes, they should return the same results. Put another way, a configuration file that enumerates DNS servers isn't the appropriate place to "merge" things. I don't have a strong position on that argument. Just trying to state it more clearly.
- wrs 9y ago
  If you limit requests to the different servers to the domains they cover, that's fine (various resolvers do this in different ways). If you tell the resolver they're all equivalent, they need to all be equivalent.
  - joombaga 9y ago
    Understood; thanks for clarifying.
- jasomill 9y ago
  No, one should probably never use unregistered suffixes. But that’s not to say that all DNS names should be public, nor that NAT is practically avoidable, so this is beside the point. What one should do is only choose DNS servers capable of properly resolving all addresses, as it shouldn’t be the resolver’s responsibility to “shop around” for answers. Fixing VPN configurations that complicate this should be the responsibility of VPN client software — by providing a forwarding DNS server that falls back to the originally configured servers, say — not the resolver, and similarly for any other cases where standard resolver behavior does not suffice. Application software, for its part, shouldn’t generally rely on implementation-defined resolver behavior, as doing so is obviously not portable, but that’s less sinful than a VPN relying on client library-level support to work around the fact that it’s exposing a broken network configuration. Note that I have no axe to grind one way or another in the systemd debate: Linux isn’t me primary OS, but I regularly use and abuse both systemd and non-systemd Linux systems, and all the Linux init systems seem to work at least as well as any other system I regularly use (OS X, FreeBSD, Windows) for most practical purposes.
AceJohnny2 9y ago
> These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. So much for CDNs or internal networks then.
- lolc 9y ago
  > all DNS servers [in resolv.conf] are expected to return the same results Better?
  - AceJohnny2 9y ago
    Yes, actually :) That's a crucial distinction.
kazinator 9y ago
Not all DNS servers will return the result in the same amount of time and same amount of network/computing resources. If the ordder is predictable, I can do some shadowing: if A is searched before B, I can put some entry into A which blocks certain sites or redirects. Even if we just have one server A, there is still an order! The recursive DNS search implements that order. Each server in the chain has the option to return its own local result, which doesn't have to coincide with the next server that is used. So, that is to say, DNS is implicitly ordered from the leaf server you're using up to the root one.
- JdeBP 9y ago
  The resolving proxy server that one sends queries to, from a client library (built into the C library or otherwise), is not a "leaf". This is the same erroneous mental model of the DNS that got the world the completely wrongheaded tool known as dnstracer, that bamboozled people for years. * http://jdebp.eu./FGA/dnstracer-incorrect-algorithm.html http://jdebp.eu./FGA/dnstracer-incorrect-algorithm.html There is not a "chain" of servers, each passing on queries to the next one along. There is a resolving proxy DNS server that takes a front-end query, makes a bunch of back-end query+response transactions with content DNS servers, and stiches their results together to make the final front-end response. * http://jdebp.eu./FGA/dns-query-resolution.html http://jdebp.eu./FGA/dns-query-resolution.html The aforementioned DNS client library can be given the IP addresses of multiple such resolving proxy DNS servers, but there has not been one single conventional order of querying them for far longer than systemd has been around. Different DNS client libraries do it in different ways. nslookup's internal DNS client library does something different to the ISC DNS client library that forms part of many C libraries, as I have already pointed out in this discussion. The whole "chain of servers" model is one of the commonest misunderstandings of the DNS, in my personal fairly long experience of people with DNS problems. It is quite wrong.
- adrianmonk 9y ago
  The pre-systemd behavior is that the order is not really completely predictable. First of all, DNS queries are usually made with UDP. There is no ack or retransmission with UDP, and the network stack is free to drop messages if necessary, such as if a queue fills up or the network is congested. (Hence it has the nickname "Unreliable Datagram Protocol".) Second, "man resolv.conf" has this to say about what happens when you have multiple DNS servers listed: "The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made." Consider what happens when you combine those two together. You roll the dice and send a UDP request to the first server. Then it rolls the dice by sending a UDP response to you. If that doesn't work, you send to the second server. If that doesn't work, then you try the first server again. TLDR: In the case of lost packets, which you must assume happen because it's UDP, it alternates between servers. The only thing that's guaranteed is which one comes up in the rotation first.
Crontab 9y ago
> all DNS servers are expected to return the same results Is that assumption true anymore, given a world where some ISPs issue redirection for unfindable domain name[1]? [1] http://www.dslreports.com/shownews/103386 http://www.dslreports.com/shownews/103386
- joombaga 9y ago
  I think it's, perhaps, a faulty assumption, but a correct expectation, and ISPs that issue redirection for unresolvable domains have in fact misconfigured their DNS servers. EDIT: Also, Comcast shut down their Domain Helper as it is fundamentally incompatible with DNSSEC http://corporate.comcast.com/comcast-voices/comcast-domain-helper-shuts-down http://corporate.comcast.com/comcast-voices/comcast-domain-h...
  - Crontab 9y ago
    I hadn't heard about that. Thanks for the info.

by wrs 9y ago

kelnos 9y ago

Real-world usage, obviously. If you're going to replace a foundational component of the OS, you need to replace it in a bug-compatible way, period. No compromises. A common use for this behavior is VPNs: you sign onto a VPN, and the VPN's DNS server gets prepended to the list in resolv.conf, which allows you to resolve internal addresses (and still fall back to your regular DNS). This is a crazy-common use case, and it's incredibly arrogant of Poettering to just dismiss it out of hand, or even attempt to argue with it.

tssva 9y ago
Which real world usage? dnsmasq and Windows both follow the systemd behavior by default.
wrs 9y ago
Our VPN is configured to replace the server list, which is the correct behavior, because the new servers don't return the same answers as the old ones. So again, those other VPNs are misconfigured.
geofft 9y ago
On macOS, there's a straightforward solution for the VPN use case that doesn't involve a "fall back": the VPN registers its DNS servers for specific domain names, and so anything in example.com has its list of DNS servers (tunneled through the VPN) and everything else has the original list of DNS servers from your physical network connection. See resolver(5) https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man5/resolver.5.html https://developer.apple.com/legacy/library/documentation/Dar... for a description of doing this with files named like "/etc/resolver/example.com" (and having the same syntax as resolv.conf), but you can also do this with SystemConfiguration (googling "SupplementalMatchDomains" should be helpful). It looks like systemd-resolved implements something similar: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net... The fact that there is an incredibly common use case that consists of an unreliable workaround for people who don't have a local resolver that does this correctly is hardly reason to avoid writing a local resolver that works correctly.
- kelnos 9y ago
  If you have an incredibly simple internal network, that might work fine. It certainly would not work at my organization.

subway 9y ago

No, this is not a misconfiguration. It is a configuration that expects the resolver to behave according to the algorithm described in resolver(5). systemd chose to wedge itself in as a resolver, then broke that contract.

jerf 9y ago
I was curious about this, since it sort of impacts my wourld, and decided to check this. There's a RES_ROTATE flag in there, described as "Causes round-robin selection of name servers from among those listed. This has the effect of spreading the query load among all listed servers, rather than having all clients try the first listed server first every time." Claim checks out.
wrs 9y ago
resolver(5) describes the behavior of glibc, not DNS or systemd. Systemd will read resolv.conf if you don't want to configure resolved.conf directly, but that's a convenience, not a contract. Dnsmasq also reads resolv.conf for convenience, and puts yet another interpretation on it (which will also break this use case). Both systemd and dnsmasq expect that you'll tell glibc to just ask the local resolver (them).

joombaga 9y ago

> all DNS servers are expected to return the same result Really? My local DNS server should return the same results as 8.8.8.8? Should one never use unregistered suffixes?

JdeBP 9y ago
All listed proxy DNS servers should provide the same view of the DNS namespace. If your local proxy DNS server provides a different view to Google Public DNS, then it is wrong to list them together. * http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-same-service.html http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-sam... And no, you shouldn't just blithely assume that you own parts of the DNS namespace that you do not, either. There are plenty of other people's mistakes to learn from here. Don't repeat them. * http://jdebp.eu./FGA/dns-use-domain-names-that-you-own.html http://jdebp.eu./FGA/dns-use-domain-names-that-you-own.html
rosser 9y ago
I think the argument is that if you do put both your local DNS server and 8.8.8.8 in your /etc/resolv.conf, then yes, they should return the same results. Put another way, a configuration file that enumerates DNS servers isn't the appropriate place to "merge" things. I don't have a strong position on that argument. Just trying to state it more clearly.
wrs 9y ago
If you limit requests to the different servers to the domains they cover, that's fine (various resolvers do this in different ways). If you tell the resolver they're all equivalent, they need to all be equivalent.
- joombaga 9y ago
  Understood; thanks for clarifying.
jasomill 9y ago
No, one should probably never use unregistered suffixes. But that’s not to say that all DNS names should be public, nor that NAT is practically avoidable, so this is beside the point. What one should do is only choose DNS servers capable of properly resolving all addresses, as it shouldn’t be the resolver’s responsibility to “shop around” for answers. Fixing VPN configurations that complicate this should be the responsibility of VPN client software — by providing a forwarding DNS server that falls back to the originally configured servers, say — not the resolver, and similarly for any other cases where standard resolver behavior does not suffice. Application software, for its part, shouldn’t generally rely on implementation-defined resolver behavior, as doing so is obviously not portable, but that’s less sinful than a VPN relying on client library-level support to work around the fact that it’s exposing a broken network configuration. Note that I have no axe to grind one way or another in the systemd debate: Linux isn’t me primary OS, but I regularly use and abuse both systemd and non-systemd Linux systems, and all the Linux init systems seem to work at least as well as any other system I regularly use (OS X, FreeBSD, Windows) for most practical purposes.

AceJohnny2 9y ago

> These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. So much for CDNs or internal networks then.

lolc 9y ago
> all DNS servers [in resolv.conf] are expected to return the same results Better?
- AceJohnny2 9y ago
  Yes, actually :) That's a crucial distinction.

kazinator 9y ago

Not all DNS servers will return the result in the same amount of time and same amount of network/computing resources. If the ordder is predictable, I can do some shadowing: if A is searched before B, I can put some entry into A which blocks certain sites or redirects. Even if we just have one server A, there is still an order! The recursive DNS search implements that order. Each server in the chain has the option to return its own local result, which doesn't have to coincide with the next server that is used. So, that is to say, DNS is implicitly ordered from the leaf server you're using up to the root one.

JdeBP 9y ago
The resolving proxy server that one sends queries to, from a client library (built into the C library or otherwise), is not a "leaf". This is the same erroneous mental model of the DNS that got the world the completely wrongheaded tool known as dnstracer, that bamboozled people for years. * http://jdebp.eu./FGA/dnstracer-incorrect-algorithm.html http://jdebp.eu./FGA/dnstracer-incorrect-algorithm.html There is not a "chain" of servers, each passing on queries to the next one along. There is a resolving proxy DNS server that takes a front-end query, makes a bunch of back-end query+response transactions with content DNS servers, and stiches their results together to make the final front-end response. * http://jdebp.eu./FGA/dns-query-resolution.html http://jdebp.eu./FGA/dns-query-resolution.html The aforementioned DNS client library can be given the IP addresses of multiple such resolving proxy DNS servers, but there has not been one single conventional order of querying them for far longer than systemd has been around. Different DNS client libraries do it in different ways. nslookup's internal DNS client library does something different to the ISC DNS client library that forms part of many C libraries, as I have already pointed out in this discussion. The whole "chain of servers" model is one of the commonest misunderstandings of the DNS, in my personal fairly long experience of people with DNS problems. It is quite wrong.
adrianmonk 9y ago
The pre-systemd behavior is that the order is not really completely predictable. First of all, DNS queries are usually made with UDP. There is no ack or retransmission with UDP, and the network stack is free to drop messages if necessary, such as if a queue fills up or the network is congested. (Hence it has the nickname "Unreliable Datagram Protocol".) Second, "man resolv.conf" has this to say about what happens when you have multiple DNS servers listed: "The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made." Consider what happens when you combine those two together. You roll the dice and send a UDP request to the first server. Then it rolls the dice by sending a UDP response to you. If that doesn't work, you send to the second server. If that doesn't work, then you try the first server again. TLDR: In the case of lost packets, which you must assume happen because it's UDP, it alternates between servers. The only thing that's guaranteed is which one comes up in the rotation first.

Crontab 9y ago

> all DNS servers are expected to return the same results Is that assumption true anymore, given a world where some ISPs issue redirection for unfindable domain name[1]? [1] http://www.dslreports.com/shownews/103386 http://www.dslreports.com/shownews/103386

joombaga 9y ago
I think it's, perhaps, a faulty assumption, but a correct expectation, and ISPs that issue redirection for unresolvable domains have in fact misconfigured their DNS servers. EDIT: Also, Comcast shut down their Domain Helper as it is fundamentally incompatible with DNSSEC http://corporate.comcast.com/comcast-voices/comcast-domain-helper-shuts-down http://corporate.comcast.com/comcast-voices/comcast-domain-h...
- Crontab 9y ago
  I hadn't heard about that. Thanks for the info.