14 ms·

Systemd-resolved does not keep the order of the DNS servers

89 points by beefhash 9y ago

mikelward 9y ago
dnsmasq does something similar by default. http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2009q3/003295.html http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/200...
- mjw1007 9y ago
  Right, and (many years ago) they added automatic "failback", as well as the "failover" behaviour that systemd-resolved has. So this isn't a case of systemd-resolved deciding to arbitrarily do things in a new and broken way; rather it's systemd-resolved still being rather immature (but for some reason already ending up as the default on lots of people's systems).
yarrel 9y ago
Why does an init system do DNS lookup? And in such a breaking way?
- thehardsphere 9y ago
  Because it's not merely an init system anymore. That was the initial pitch for it; now it controls everything, including things that were not broken to begin with.
- tetromino_ 9y ago
  > Why does an init system do DNS lookup? For caching and DNSSEC validation at the system level instead of needing to be handled ad hoc by each individual framework or application. > And in such a breaking way? See https://github.com/systemd/systemd/issues/5755#issuecomment-315810471 https://github.com/systemd/systemd/issues/5755#issuecomment-... Systemd designers optimized their new resolver's behavior for one common use case, but apparently never considered that it breaks another (undocumented) use case. That seems like an example of a lack of real user testing for an important feature before cutting a new release.
  - Karunamon 9y ago
    >For caching and DNSSEC validation at the system level instead of needing to be handled ad hoc by each individual framework or application. Given the hostility and generally crappy attitude of the systemd's maintainers, this reads to me as "Reinventing the wheel and throwing out decades of knowledge because shiny is better".
  - protomyth 9y ago
    How is it undocumented? The resolv.conf man page on Red Hat says "If there are multiple servers, the resolver library queries them in the order listed."
- jcelerier 9y ago
  systemd-resolved is not an init system, it's a separate software.
  - Karunamon 9y ago
    Rant: Systemd as a project run by people abuses this confusion. In the minds of most normal people, when you prefix a name with another name, you're implying a deep association. Apple Watch, Apple iPhone. Systemd Journal, Systemd resolver. When people criticize Systemd's continued absorption of Linux system services, they point at it being "modular" (despite it being a rapidly moving target with no real alternative replacements - i.e. modular in the purely theoretical sense of the word), and then take criticisms of the individual parts as criticisms of the whole in an attempt to shout the detractors down strawman-style. Most of the problems with systemd can be chalked up to its leadership and utterly toxic cultural impact, rather than its technical merits (or lack thereof). Most people with a problem with systemd don't have a problem with systemd as code - they have a problem with systemd as this creeping monstrosity that keeps eating services that have worked fine for decades in the name of objectively worse replacements. Did name resolution, NTP, and so forth really gain anything tangibly meaningful from their assimilation, other than integration with systemd? Did those services gain more than they lost in annoyances by system admins troubleshooting the changes? We arguably needed a new init system (and I'll just gloss over things like Upstart and OpenRC). We did not need yet another new NTP, DNS, cron, logger, auditd, and fucking QR code library tied into the system internals!
    - 5ilv3r 9y ago
      Well said sir!
    - jryan49 9y ago
      Linux distro maintainers are making the decision to use the systemd replacements. If you disagree, use a Linux distro that doesn't, or make your own.
      - Karunamon 9y ago
thehardsphere 9y ago
I stopped using Linux for personal before this systemd thing was an issue, but I was considering it again now that I might get a new computer and because I've got an app running on a pre-systemd Linux VPN. Since I have very little first hand experience with systemd, I have some questions. 1. Is it actually that bad for a desktop? 2. Is it actually that bad for a server? 3. If yes to either, how suitable are the alternatives (e.g. Gentoo, Slack, or BSD)?
- NikolaeVarius 9y ago
  If you have to ask, you probably don't care. If it was that bad for the average consumer, the major mainline distros wouldn't have all adopted them.
  - thehardsphere 9y ago
    I don't care today; I don't have a daily driver using this stuff today. I care for my future self who might have to deal with this. It was my understanding that the major mainline distros adopted it because Red Hat coerced Gnome into adopting it, and everyone feels the need to support Gnome.
    - jle17 9y ago
      Systemd was a personal project of Lennart, it didn't even originally came from Red Hat. Red Hat certainly hasn't the power to coerce Gnome into anything. And major distros all had long discussions and debates on public mailing lists where the details of why they adopted it is discussed.
    - makomk 9y ago
      Pretty much. Originally, the Gnome developers told the distros that it wasn't a big deal because logind would run just fine with whatever init system they chose to use. The a few years later once this was deeply embedded in everything Poettering declared that this was never supported, the distros shouldn't have done it, and it would break in the imminent future. The major distros all switched in the next release whether they wanted to or not.
king_phil 9y ago
I really, really like Linus Torvalds idea for the kernel: "no regressions". NO. REGRESSIONS. I have no idea what the problem of Lennart is.
- gaius 9y ago
  The problem is the supposed adult supervision at Red Hat the corporate entity, that is funding all of this
  - isostatic 9y ago
    Doesn't explain why Debian and derivatives followed suit.
    - 5ilv3r 9y ago
      They follow freedesktop.
- deleted 9y ago
  [deleted]
wrs 9y ago
These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. But their misconfiguration happens to work most of the time for their use case. The solution is either to implement a "bug-compatible" priority flag in resolved, or to make that a first-class use case with its own flag ("emergency backup nameserver"?). It's a classic case...should the software follow the spec, or the real-world usage? And people complain either way.
- kelnos 9y ago
  Real-world usage, obviously. If you're going to replace a foundational component of the OS, you need to replace it in a bug-compatible way, period. No compromises. A common use for this behavior is VPNs: you sign onto a VPN, and the VPN's DNS server gets prepended to the list in resolv.conf, which allows you to resolve internal addresses (and still fall back to your regular DNS). This is a crazy-common use case, and it's incredibly arrogant of Poettering to just dismiss it out of hand, or even attempt to argue with it.
  - tssva 9y ago
    Which real world usage? dnsmasq and Windows both follow the systemd behavior by default.
  - wrs 9y ago
    Our VPN is configured to replace the server list, which is the correct behavior, because the new servers don't return the same answers as the old ones. So again, those other VPNs are misconfigured.
  - geofft 9y ago
    On macOS, there's a straightforward solution for the VPN use case that doesn't involve a "fall back": the VPN registers its DNS servers for specific domain names, and so anything in example.com has its list of DNS servers (tunneled through the VPN) and everything else has the original list of DNS servers from your physical network connection. See resolver(5) https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man5/resolver.5.html https://developer.apple.com/legacy/library/documentation/Dar... for a description of doing this with files named like "/etc/resolver/example.com" (and having the same syntax as resolv.conf), but you can also do this with SystemConfiguration (googling "SupplementalMatchDomains" should be helpful). It looks like systemd-resolved implements something similar: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net... The fact that there is an incredibly common use case that consists of an unreliable workaround for people who don't have a local resolver that does this correctly is hardly reason to avoid writing a local resolver that works correctly.
ealexhudson 9y ago
I would have to side with systemd project here. I had never heard of the ordering of the DNS servers being meaningful, and I'm amazed people are relying on that behaviour described in the ticket. systemd appears to be the only project where getting rid of legacy cruft is seen as a bad thing. Having a DNS setup that works in a different way when stuff fails should be seen as a bug; DNS is supposed to be reliable.
- kazinator 9y ago
  > I had never heard of the ordering of the DNS servers being meaningful Well, congratulations, there is a first time to have heard everything. Ordering of DNS servers matters. For instance, you might have a fast one near-by (perhaps on the same machine or LAN segment); and a fallback one that is more costly to use (few hops of latency, more network resources). > DNS is supposed to be reliable. In that case, what you should never have heard of is people pointing to more than one DNS server simultaneously. They should have the one and only one which is reliable and that's it. (In fact, that is not an uncommon scenario; that DNS server does the recursion to others if it doesn't have the info.)
  - jle17 9y ago
    > Ordering of DNS servers matters. For instance, you might have a fast one near-by (perhaps on the same machine or LAN segment); and a fallback one that is more costly to use (few hops of latency, more network resources). Yes, in terms of performance it's pretty common that the order matters, but the bug report is about the order mattering in terms of result. This is less common and seems like a bad idea.
    - kazinator 9y ago
      Clearly, if for any given result, server A must assert its answer over server B, there cannot be a simple timeout or failure fallback from A to B; only if A indicates that it is in working order, but declines to provide an answer.
  - amluto 9y ago
stonogo 9y ago
The behavior these people are asking for is an implementation detail specific to glibc. It's also incorrect: all DNS resolvers listed in resolv.conf are assumed by relevant standards to return the same results as each other. I feel bad for the systemd developers, getting bug reports from people who don't understand their own computers.
- king_phil 9y ago
  Can you point out the relevant RFC, please? Didn't know about that.
  - bluejekyll 9y ago
    I don't believe resolv.conf is in any DNS RFC individually. Here's a list of all RFCs related to DNS: https://www.isc.org/community/rfcs/dns/ https://www.isc.org/community/rfcs/dns/ In https://tools.ietf.org/html/rfc1034 https://tools.ietf.org/html/rfc1034 this is discussed: The resolver always starts with a list of server names to query (SLIST). This list will be all NS RRs which correspond to the nearest ancestor zone that the resolver knows about. To avoid startup problems, the resolver should have a set of default servers which it will ask should it have no current NS RRs which are appropriate. The resolver then adds to SLIST all of the known addresses for the name servers, and may start parallel requests to acquire the addresses of the servers when the resolver has the name, but no addresses, for the name servers. I believe it would be an accurate statement that a DNS resolver which is incapable of servicing all DNS queries by the client is something which many clients are not designed to expect. But relying on the behavior of glibc in the way it attempts lookups is in essence relying on the internal workings of that software rather than anything written in the spec. BIND9 supports 'views' (iirc) which allow for splitting queries between internal vs. external DNS. But that would be the resolver as opposed the users desktop implementing that.
  - stonogo 9y ago
    From RFC 1034: "The strategy is to cycle around all of the addresses for all of the servers with a timeout between each transmission." Note that it does not specify an order -- the resolver determines that, and RFC 1035 even recommends keeping a weighted list of nameservers prioritized by response time: "To complete initialization of SLIST, the resolver attaches whatever history information it has to the each address in SLIST. This will usually consist of some sort of weighted averages for the response time of the address, and the batting average of the address (i.e., how often the address responded at all to the request)." ...which is exactly what systemd is doing. The glibc behavior is a naive simplest-implementation approach, and if your infrastructure depends on an implementation detail of a specific libc resolver, you should install proper configurable resolvers and use those instead. The fact that all nameservers are expected to return identical results is inherent in the design of DNS. It's one namespace, and if you need to split it, the client resolver is the wrong place to do that too.
bahjoite 9y ago
> The "Switching to fallback DNS server 8.8.8.8." message indicates that you have no DNS servers configured at all, in which case resolved will use compiled-in fallback servers because it tries hard to just work also if you have a locally misconfigured system It's good to have the knowledge that Google servers are compiled-in to resolved.
- geofft 9y ago
  They're "compiled in" in the sense that they're the upstream default - people are generally expected to use systemd from a Linux distro, and distributors are expected to patch this to some appropriate default (or patch out the fallback entirely, if your distro would prefer your users get no DNS to Google DNS, which seems like a defensible decision to make). There are no upstream binary releases. 8.8.8.8 is as reasonable as anything else for someone running `make` on the upstream sources and hacking on things; it's probably not a great default for a downstream distributor.
  - bahjoite 9y ago
    The relevant option to change is "dns-servers" at https://github.com/systemd/systemd/blob/master/meson_options.txt#L160 https://github.com/systemd/systemd/blob/master/meson_options...
    - hedora 9y ago
      Heh. The line above the one you highlight makes dnssec vulnerable to MITM attacks by default. Nice. (Also, the one below points at more Google infrastructure...)
    - TheDong 9y ago
      That's a link to master. For future reference, here's a stable link which will not rot away over time: https://github.com/systemd/systemd/blob/v234/meson_options.txt#L156-L158 https://github.com/systemd/systemd/blob/v234/meson_options.t... Already the line numbers are slightly different
  - deleted 9y ago
binaryapparatus 9y ago
This is also very likely to cause DNS leaks while using VPN. Which kind of makes whole idea of VPN unreliable and/or dangerous to rely on. Not because VPN design has anything to do with this security hole but systemd tries hard to do stuff it doesn't really understand.
- geofft 9y ago
  Quite the opposite. The system of expecting resolver order to be respected (the glibc behavior) is the one that is likely cause DNS leaks while using a VPN: if your VPN connection or its DNS server is intermittently flaky, glibc will gladly go try your next, public resolver with the internal domain name. Meanwhile, systemd has actual functionality for scoping your VPN domain names to your VPN's DNS server only: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net...
  - binaryapparatus 9y ago
    Regardless of marketing pitch, resolver is no job for systemd. Or any other init/pid 1 process. Systemd is known to but in into areas that are not / should not be of concern for it. Also known to do stuff half baked or changing things because you can. So I'd very much love to see systemd fingers completely outside of dns resolver, that way we wouldn't have this conversation at all.
    - geofft 9y ago
      > Regardless of marketing pitch, resolver is no job for systemd. Or any other init/pid 1 process. Correct, which is why it's not part of /bin/systemd or any other init/pid 1 process.
      - binaryapparatus 9y ago
        You know, first thing that raised some flags with systemd for me, couple of years ago, is constant marketing push, like this one. Having one monolith executable or ten executables that increasingly demand each other is the same thing. Oh wait its name also suggests it is part of the package, being named Systemd-resolved. There is number of reality distortions like this one that systemd people tried to perform, during the several past years.
js2 9y ago
systemd's behavior is completely reasonable. The folks arguing otherwise on the issue would probably be better served by installing dnsmasq locally to do what they want and pointing systemd at that. It may be worth pointing out that dnsmasq has a similar behavior by default and points out the correct way to override specific hostnames/zones: Q: My company's nameserver knows about some names which aren't in the public DNS. Even though I put it first in /etc/resolv.conf, it doesn't work: dnsmasq seems not to use the nameservers in the order given. What am I doing wrong? A: By default, dnsmasq treats all the nameservers it knows about as equal: it picks the one to use using an algorithm designed to avoid nameservers which aren't responding. To make dnsmasq use the servers in order, give it the -o flag. If you want some queries sent to a special server, think about using the -S flag to give the IP address of that server, and telling dnsmasq exactly which domains to use the server for.
cesarb 9y ago
Putting servers with different views of the DNS in resolv.conf seems unreliable to me, even with glibc's behavior. If the first server "glitches" for any reason, the process will get the response from the second server, which might not be what you wanted. And worse, some processes (like browsers) have their own cache of DNS responses, so the incorrect response might get cached. That is: it was already broken, systemd-resolvd just made the breakage more visible. (If I understood it correctly, it only switches after the first "glitch", so the only difference is that it doesn't switch back until the second server also "glitches".)
5ilv3r 9y ago
I run split horizon DNS and have my local DNS server listed before my ISP's in my DHCP server configuration. The order (to me) is important. I want to hit MY dns server always (which does cache), unless it is unavailable, and then I want to hit my ISPs as a fallback only. Is there a suggested way to do this?
- yjftsjthsd-h 9y ago
  Just from this thread, several: With dnsmasq: https://news.ycombinator.com/item?id=15231319 https://news.ycombinator.com/item?id=15231319 With systemd: https://news.ycombinator.com/item?id=15232027 https://news.ycombinator.com/item?id=15232027
  - 5ilv3r 9y ago
    Thanks for the reply, but can you see how the very simple configuration possible on the DHCP server suddenly becomes much much more complicated with the above solutions? You're basically suggesting I run a chef server (or some other central workstation management) in my home to allow talking to machines on my local domain by hostnames. That's tough to swallow.
    - comex 9y ago
      Is your setup actually likely to run into situations where the DNS server is unavailable but Internet access is otherwise working? Basic home setups usually have DNS running (if at all) on the same device (router) that’s also acting as a gateway, so such a selective outage would be unlikely except in the event of misconfiguration - and in that case I’d want my setup to fail hard, personally, so I’d know to fix it, rather than having it silently slow down DNS lookups. Thus I wouldn’t intentionally set a public DNS server as an alternative to the primary one on the router. But maybe your setup is more complex, or you have a use case I’m not thinking of...
      - 5ilv3r 9y ago
        I run DNS on my file server since it's on all the time anyway. It must fail gracefully or the wife approval factor goes down.
upofadown 9y ago
FWIW, the OpenBSD resolve.conf(5) man page says: >If there are multiple servers, the resolver library queries them in the order listed.
- chronid 9y ago
  That's the same in FreeBSD: > (The algorithm used is to try a name server, and if the query times out, trythe next, until out of name servers, then repeat trying allthe name servers until a maximum number of retries are made). So it seems there is a de facto standard for the default libraries in some *NIX OSes?
  - feld 9y ago
    Confirmed in Solaris 10 and Solaris 11 as well.
  - JdeBP 9y ago
    No. All that it is is common heritage in the BIND DNS client library whose doco this is. It is a widely used DNS client library that has made it into numerous operating system's C libraries, from Solaris through GNU libc to OS/2. The manual pages for that library have spread along with it. * https://www.isc.org/downloads/libbind/ https://www.isc.org/downloads/libbind/ * http://wiki.netbsd.org/individual-software-releases/netresolv/ http://wiki.netbsd.org/individual-software-releases/netresol... * https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/README;hb=4e8286acfa4224ac9ccfb07e90b8fd70fab1467e https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/READ... One can even trace its spread from, say, 386BSD through NetBSD to OpenBSD. * https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acde277d4ff9dcfd3b/share/man/man5/resolv.conf.5 https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acd... * https://github.com/openbsd/src/blob/df930be708d50e9715f173caa26ffe1b7599b157/share/man/man5/resolv.conf.5 https://github.com/openbsd/src/blob/df930be708d50e9715f173ca... Manual pages are not standards, not specifications, but implementation documentation. As Chuck McManis said in this very discussion, to misuse them as standards or specifications is to mistakenly treat implementation as architecture. The BIND DNS client library is far from the only DNS client library in existence. Other DNS client libraries read /etc/resolv.conf, as part of a BIND DNS client compatibility shim or otherwise. They do not have the same implementation details as the BIND DNS client; they do not all have the same access patterns as the BIND DNS client. Indeed, as I have mentioned elsewhere in this very discussion, even other tools from the same origin, namely nslookup, read /etc/resolv.conf and have different access patterns. * https://cr.yp.to/djbdns/blurb/library.html https://cr.yp.to/djbdns/blurb/library.html * http://www.chiark.greenend.org.uk/~ian/adns/ http://www.chiark.greenend.org.uk/~ian/adns/ * http://25thandclement.com/~william/projects/dns.c.html http://25thandclement.com/~william/projects/dns.c.html * https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tgs8nb4Rg/zLkQhY7O_zkJ https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tg... There is not a single common conventional, let alone standard, behaviour to rely upon.
jimjimjim 9y ago
remove it. i'm more and more becoming convinced that the systemd ecosystem should be considered a regression.
totony 9y ago
Shipping incompatible software is somehow an issue, but the bigger issue seems to be that canonical shipped the software without testing or ensuring it didnt break compatibility The job of ensuring compatibility of softwares before replacing them is a distribution's job
- throwawaymanbot 9y ago
  Ah yes, everyone else's fault...
throwawaymanbot 9y ago
I like to stick to real world. For example. even though on paper IIS web server is technically amazing, In the real world, no one really uses IIS as part of their web stack. (For many different reasons). Again, all I see is arrogance. Its never SystemDs heavy handed approach thats wrong, its always everyone else's fault for their implementation issues.

89 points by beefhash 9y ago

dnsmasq does something similar by default. http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2009q3/003295.html http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/200...

mjw1007 9y ago
Right, and (many years ago) they added automatic "failback", as well as the "failover" behaviour that systemd-resolved has. So this isn't a case of systemd-resolved deciding to arbitrarily do things in a new and broken way; rather it's systemd-resolved still being rather immature (but for some reason already ending up as the default on lots of people's systems).

yarrel 9y ago

Why does an init system do DNS lookup? And in such a breaking way?

thehardsphere 9y ago
Because it's not merely an init system anymore. That was the initial pitch for it; now it controls everything, including things that were not broken to begin with.
tetromino_ 9y ago
> Why does an init system do DNS lookup? For caching and DNSSEC validation at the system level instead of needing to be handled ad hoc by each individual framework or application. > And in such a breaking way? See https://github.com/systemd/systemd/issues/5755#issuecomment-315810471 https://github.com/systemd/systemd/issues/5755#issuecomment-... Systemd designers optimized their new resolver's behavior for one common use case, but apparently never considered that it breaks another (undocumented) use case. That seems like an example of a lack of real user testing for an important feature before cutting a new release.
- Karunamon 9y ago
  >For caching and DNSSEC validation at the system level instead of needing to be handled ad hoc by each individual framework or application. Given the hostility and generally crappy attitude of the systemd's maintainers, this reads to me as "Reinventing the wheel and throwing out decades of knowledge because shiny is better".
- protomyth 9y ago
  How is it undocumented? The resolv.conf man page on Red Hat says "If there are multiple servers, the resolver library queries them in the order listed."
jcelerier 9y ago
systemd-resolved is not an init system, it's a separate software.
- Karunamon 9y ago
  Rant: Systemd as a project run by people abuses this confusion. In the minds of most normal people, when you prefix a name with another name, you're implying a deep association. Apple Watch, Apple iPhone. Systemd Journal, Systemd resolver. When people criticize Systemd's continued absorption of Linux system services, they point at it being "modular" (despite it being a rapidly moving target with no real alternative replacements - i.e. modular in the purely theoretical sense of the word), and then take criticisms of the individual parts as criticisms of the whole in an attempt to shout the detractors down strawman-style. Most of the problems with systemd can be chalked up to its leadership and utterly toxic cultural impact, rather than its technical merits (or lack thereof). Most people with a problem with systemd don't have a problem with systemd as code - they have a problem with systemd as this creeping monstrosity that keeps eating services that have worked fine for decades in the name of objectively worse replacements. Did name resolution, NTP, and so forth really gain anything tangibly meaningful from their assimilation, other than integration with systemd? Did those services gain more than they lost in annoyances by system admins troubleshooting the changes? We arguably needed a new init system (and I'll just gloss over things like Upstart and OpenRC). We did not need yet another new NTP, DNS, cron, logger, auditd, and fucking QR code library tied into the system internals!
  - 5ilv3r 9y ago
    Well said sir!
  - jryan49 9y ago
    Linux distro maintainers are making the decision to use the systemd replacements. If you disagree, use a Linux distro that doesn't, or make your own.
    - Karunamon 9y ago

thehardsphere 9y ago

I stopped using Linux for personal before this systemd thing was an issue, but I was considering it again now that I might get a new computer and because I've got an app running on a pre-systemd Linux VPN. Since I have very little first hand experience with systemd, I have some questions. 1. Is it actually that bad for a desktop? 2. Is it actually that bad for a server? 3. If yes to either, how suitable are the alternatives (e.g. Gentoo, Slack, or BSD)?

NikolaeVarius 9y ago
If you have to ask, you probably don't care. If it was that bad for the average consumer, the major mainline distros wouldn't have all adopted them.
- thehardsphere 9y ago
  I don't care today; I don't have a daily driver using this stuff today. I care for my future self who might have to deal with this. It was my understanding that the major mainline distros adopted it because Red Hat coerced Gnome into adopting it, and everyone feels the need to support Gnome.
  - jle17 9y ago
    Systemd was a personal project of Lennart, it didn't even originally came from Red Hat. Red Hat certainly hasn't the power to coerce Gnome into anything. And major distros all had long discussions and debates on public mailing lists where the details of why they adopted it is discussed.
  - makomk 9y ago
    Pretty much. Originally, the Gnome developers told the distros that it wasn't a big deal because logind would run just fine with whatever init system they chose to use. The a few years later once this was deeply embedded in everything Poettering declared that this was never supported, the distros shouldn't have done it, and it would break in the imminent future. The major distros all switched in the next release whether they wanted to or not.

king_phil 9y ago

I really, really like Linus Torvalds idea for the kernel: "no regressions". NO. REGRESSIONS. I have no idea what the problem of Lennart is.

gaius 9y ago
The problem is the supposed adult supervision at Red Hat the corporate entity, that is funding all of this
- isostatic 9y ago
  Doesn't explain why Debian and derivatives followed suit.
  - 5ilv3r 9y ago
    They follow freedesktop.
deleted 9y ago
[deleted]

wrs 9y ago

These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. But their misconfiguration happens to work most of the time for their use case. The solution is either to implement a "bug-compatible" priority flag in resolved, or to make that a first-class use case with its own flag ("emergency backup nameserver"?). It's a classic case...should the software follow the spec, or the real-world usage? And people complain either way.

kelnos 9y ago
Real-world usage, obviously. If you're going to replace a foundational component of the OS, you need to replace it in a bug-compatible way, period. No compromises. A common use for this behavior is VPNs: you sign onto a VPN, and the VPN's DNS server gets prepended to the list in resolv.conf, which allows you to resolve internal addresses (and still fall back to your regular DNS). This is a crazy-common use case, and it's incredibly arrogant of Poettering to just dismiss it out of hand, or even attempt to argue with it.
- tssva 9y ago
  Which real world usage? dnsmasq and Windows both follow the systemd behavior by default.
- wrs 9y ago
  Our VPN is configured to replace the server list, which is the correct behavior, because the new servers don't return the same answers as the old ones. So again, those other VPNs are misconfigured.
- geofft 9y ago
  On macOS, there's a straightforward solution for the VPN use case that doesn't involve a "fall back": the VPN registers its DNS servers for specific domain names, and so anything in example.com has its list of DNS servers (tunneled through the VPN) and everything else has the original list of DNS servers from your physical network connection. See resolver(5) https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man5/resolver.5.html https://developer.apple.com/legacy/library/documentation/Dar... for a description of doing this with files named like "/etc/resolver/example.com" (and having the same syntax as resolv.conf), but you can also do this with SystemConfiguration (googling "SupplementalMatchDomains" should be helpful). It looks like systemd-resolved implements something similar: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net... The fact that there is an incredibly common use case that consists of an unreliable workaround for people who don't have a local resolver that does this correctly is hardly reason to avoid writing a local resolver that works correctly.

ealexhudson 9y ago

I would have to side with systemd project here. I had never heard of the ordering of the DNS servers being meaningful, and I'm amazed people are relying on that behaviour described in the ticket. systemd appears to be the only project where getting rid of legacy cruft is seen as a bad thing. Having a DNS setup that works in a different way when stuff fails should be seen as a bug; DNS is supposed to be reliable.

kazinator 9y ago
> I had never heard of the ordering of the DNS servers being meaningful Well, congratulations, there is a first time to have heard everything. Ordering of DNS servers matters. For instance, you might have a fast one near-by (perhaps on the same machine or LAN segment); and a fallback one that is more costly to use (few hops of latency, more network resources). > DNS is supposed to be reliable. In that case, what you should never have heard of is people pointing to more than one DNS server simultaneously. They should have the one and only one which is reliable and that's it. (In fact, that is not an uncommon scenario; that DNS server does the recursion to others if it doesn't have the info.)
- jle17 9y ago
  > Ordering of DNS servers matters. For instance, you might have a fast one near-by (perhaps on the same machine or LAN segment); and a fallback one that is more costly to use (few hops of latency, more network resources). Yes, in terms of performance it's pretty common that the order matters, but the bug report is about the order mattering in terms of result. This is less common and seems like a bad idea.
  - kazinator 9y ago
    Clearly, if for any given result, server A must assert its answer over server B, there cannot be a simple timeout or failure fallback from A to B; only if A indicates that it is in working order, but declines to provide an answer.
- amluto 9y ago

stonogo 9y ago

The behavior these people are asking for is an implementation detail specific to glibc. It's also incorrect: all DNS resolvers listed in resolv.conf are assumed by relevant standards to return the same results as each other. I feel bad for the systemd developers, getting bug reports from people who don't understand their own computers.

king_phil 9y ago
Can you point out the relevant RFC, please? Didn't know about that.
- bluejekyll 9y ago
  I don't believe resolv.conf is in any DNS RFC individually. Here's a list of all RFCs related to DNS: https://www.isc.org/community/rfcs/dns/ https://www.isc.org/community/rfcs/dns/ In https://tools.ietf.org/html/rfc1034 https://tools.ietf.org/html/rfc1034 this is discussed: The resolver always starts with a list of server names to query (SLIST). This list will be all NS RRs which correspond to the nearest ancestor zone that the resolver knows about. To avoid startup problems, the resolver should have a set of default servers which it will ask should it have no current NS RRs which are appropriate. The resolver then adds to SLIST all of the known addresses for the name servers, and may start parallel requests to acquire the addresses of the servers when the resolver has the name, but no addresses, for the name servers. I believe it would be an accurate statement that a DNS resolver which is incapable of servicing all DNS queries by the client is something which many clients are not designed to expect. But relying on the behavior of glibc in the way it attempts lookups is in essence relying on the internal workings of that software rather than anything written in the spec. BIND9 supports 'views' (iirc) which allow for splitting queries between internal vs. external DNS. But that would be the resolver as opposed the users desktop implementing that.
- stonogo 9y ago
  From RFC 1034: "The strategy is to cycle around all of the addresses for all of the servers with a timeout between each transmission." Note that it does not specify an order -- the resolver determines that, and RFC 1035 even recommends keeping a weighted list of nameservers prioritized by response time: "To complete initialization of SLIST, the resolver attaches whatever history information it has to the each address in SLIST. This will usually consist of some sort of weighted averages for the response time of the address, and the batting average of the address (i.e., how often the address responded at all to the request)." ...which is exactly what systemd is doing. The glibc behavior is a naive simplest-implementation approach, and if your infrastructure depends on an implementation detail of a specific libc resolver, you should install proper configurable resolvers and use those instead. The fact that all nameservers are expected to return identical results is inherent in the design of DNS. It's one namespace, and if you need to split it, the client resolver is the wrong place to do that too.

bahjoite 9y ago

> The "Switching to fallback DNS server 8.8.8.8." message indicates that you have no DNS servers configured at all, in which case resolved will use compiled-in fallback servers because it tries hard to just work also if you have a locally misconfigured system It's good to have the knowledge that Google servers are compiled-in to resolved.

geofft 9y ago
They're "compiled in" in the sense that they're the upstream default - people are generally expected to use systemd from a Linux distro, and distributors are expected to patch this to some appropriate default (or patch out the fallback entirely, if your distro would prefer your users get no DNS to Google DNS, which seems like a defensible decision to make). There are no upstream binary releases. 8.8.8.8 is as reasonable as anything else for someone running `make` on the upstream sources and hacking on things; it's probably not a great default for a downstream distributor.
- bahjoite 9y ago
  The relevant option to change is "dns-servers" at https://github.com/systemd/systemd/blob/master/meson_options.txt#L160 https://github.com/systemd/systemd/blob/master/meson_options...
  - hedora 9y ago
    Heh. The line above the one you highlight makes dnssec vulnerable to MITM attacks by default. Nice. (Also, the one below points at more Google infrastructure...)
  - TheDong 9y ago
    That's a link to master. For future reference, here's a stable link which will not rot away over time: https://github.com/systemd/systemd/blob/v234/meson_options.txt#L156-L158 https://github.com/systemd/systemd/blob/v234/meson_options.t... Already the line numbers are slightly different
- deleted 9y ago

binaryapparatus 9y ago

This is also very likely to cause DNS leaks while using VPN. Which kind of makes whole idea of VPN unreliable and/or dangerous to rely on. Not because VPN design has anything to do with this security hole but systemd tries hard to do stuff it doesn't really understand.

geofft 9y ago
Quite the opposite. The system of expecting resolver order to be respected (the glibc behavior) is the one that is likely cause DNS leaks while using a VPN: if your VPN connection or its DNS server is intermittently flaky, glibc will gladly go try your next, public resolver with the internal domain name. Meanwhile, systemd has actual functionality for scoping your VPN domain names to your VPN's DNS server only: https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains= https://www.freedesktop.org/software/systemd/man/systemd.net...
- binaryapparatus 9y ago
  Regardless of marketing pitch, resolver is no job for systemd. Or any other init/pid 1 process. Systemd is known to but in into areas that are not / should not be of concern for it. Also known to do stuff half baked or changing things because you can. So I'd very much love to see systemd fingers completely outside of dns resolver, that way we wouldn't have this conversation at all.
  - geofft 9y ago
    > Regardless of marketing pitch, resolver is no job for systemd. Or any other init/pid 1 process. Correct, which is why it's not part of /bin/systemd or any other init/pid 1 process.
    - binaryapparatus 9y ago
      You know, first thing that raised some flags with systemd for me, couple of years ago, is constant marketing push, like this one. Having one monolith executable or ten executables that increasingly demand each other is the same thing. Oh wait its name also suggests it is part of the package, being named Systemd-resolved. There is number of reality distortions like this one that systemd people tried to perform, during the several past years.

js2 9y ago

systemd's behavior is completely reasonable. The folks arguing otherwise on the issue would probably be better served by installing dnsmasq locally to do what they want and pointing systemd at that. It may be worth pointing out that dnsmasq has a similar behavior by default and points out the correct way to override specific hostnames/zones: Q: My company's nameserver knows about some names which aren't in the public DNS. Even though I put it first in /etc/resolv.conf, it doesn't work: dnsmasq seems not to use the nameservers in the order given. What am I doing wrong? A: By default, dnsmasq treats all the nameservers it knows about as equal: it picks the one to use using an algorithm designed to avoid nameservers which aren't responding. To make dnsmasq use the servers in order, give it the -o flag. If you want some queries sent to a special server, think about using the -S flag to give the IP address of that server, and telling dnsmasq exactly which domains to use the server for.

cesarb 9y ago

Putting servers with different views of the DNS in resolv.conf seems unreliable to me, even with glibc's behavior. If the first server "glitches" for any reason, the process will get the response from the second server, which might not be what you wanted. And worse, some processes (like browsers) have their own cache of DNS responses, so the incorrect response might get cached. That is: it was already broken, systemd-resolvd just made the breakage more visible. (If I understood it correctly, it only switches after the first "glitch", so the only difference is that it doesn't switch back until the second server also "glitches".)

5ilv3r 9y ago

I run split horizon DNS and have my local DNS server listed before my ISP's in my DHCP server configuration. The order (to me) is important. I want to hit MY dns server always (which does cache), unless it is unavailable, and then I want to hit my ISPs as a fallback only. Is there a suggested way to do this?

yjftsjthsd-h 9y ago
Just from this thread, several: With dnsmasq: https://news.ycombinator.com/item?id=15231319 https://news.ycombinator.com/item?id=15231319 With systemd: https://news.ycombinator.com/item?id=15232027 https://news.ycombinator.com/item?id=15232027
- 5ilv3r 9y ago
  Thanks for the reply, but can you see how the very simple configuration possible on the DHCP server suddenly becomes much much more complicated with the above solutions? You're basically suggesting I run a chef server (or some other central workstation management) in my home to allow talking to machines on my local domain by hostnames. That's tough to swallow.
  - comex 9y ago
    Is your setup actually likely to run into situations where the DNS server is unavailable but Internet access is otherwise working? Basic home setups usually have DNS running (if at all) on the same device (router) that’s also acting as a gateway, so such a selective outage would be unlikely except in the event of misconfiguration - and in that case I’d want my setup to fail hard, personally, so I’d know to fix it, rather than having it silently slow down DNS lookups. Thus I wouldn’t intentionally set a public DNS server as an alternative to the primary one on the router. But maybe your setup is more complex, or you have a use case I’m not thinking of...
    - 5ilv3r 9y ago
      I run DNS on my file server since it's on all the time anyway. It must fail gracefully or the wife approval factor goes down.

upofadown 9y ago

FWIW, the OpenBSD resolve.conf(5) man page says: >If there are multiple servers, the resolver library queries them in the order listed.

chronid 9y ago
That's the same in FreeBSD: > (The algorithm used is to try a name server, and if the query times out, trythe next, until out of name servers, then repeat trying allthe name servers until a maximum number of retries are made). So it seems there is a de facto standard for the default libraries in some *NIX OSes?
- feld 9y ago
  Confirmed in Solaris 10 and Solaris 11 as well.
- JdeBP 9y ago
  No. All that it is is common heritage in the BIND DNS client library whose doco this is. It is a widely used DNS client library that has made it into numerous operating system's C libraries, from Solaris through GNU libc to OS/2. The manual pages for that library have spread along with it. * https://www.isc.org/downloads/libbind/ https://www.isc.org/downloads/libbind/ * http://wiki.netbsd.org/individual-software-releases/netresolv/ http://wiki.netbsd.org/individual-software-releases/netresol... * https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/README;hb=4e8286acfa4224ac9ccfb07e90b8fd70fab1467e https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/READ... One can even trace its spread from, say, 386BSD through NetBSD to OpenBSD. * https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acde277d4ff9dcfd3b/share/man/man5/resolv.conf.5 https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acd... * https://github.com/openbsd/src/blob/df930be708d50e9715f173caa26ffe1b7599b157/share/man/man5/resolv.conf.5 https://github.com/openbsd/src/blob/df930be708d50e9715f173ca... Manual pages are not standards, not specifications, but implementation documentation. As Chuck McManis said in this very discussion, to misuse them as standards or specifications is to mistakenly treat implementation as architecture. The BIND DNS client library is far from the only DNS client library in existence. Other DNS client libraries read /etc/resolv.conf, as part of a BIND DNS client compatibility shim or otherwise. They do not have the same implementation details as the BIND DNS client; they do not all have the same access patterns as the BIND DNS client. Indeed, as I have mentioned elsewhere in this very discussion, even other tools from the same origin, namely nslookup, read /etc/resolv.conf and have different access patterns. * https://cr.yp.to/djbdns/blurb/library.html https://cr.yp.to/djbdns/blurb/library.html * http://www.chiark.greenend.org.uk/~ian/adns/ http://www.chiark.greenend.org.uk/~ian/adns/ * http://25thandclement.com/~william/projects/dns.c.html http://25thandclement.com/~william/projects/dns.c.html * https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tgs8nb4Rg/zLkQhY7O_zkJ https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tg... There is not a single common conventional, let alone standard, behaviour to rely upon.

jimjimjim 9y ago

remove it. i'm more and more becoming convinced that the systemd ecosystem should be considered a regression.

totony 9y ago

Shipping incompatible software is somehow an issue, but the bigger issue seems to be that canonical shipped the software without testing or ensuring it didnt break compatibility The job of ensuring compatibility of softwares before replacing them is a distribution's job

throwawaymanbot 9y ago
Ah yes, everyone else's fault...

throwawaymanbot 9y ago

I like to stick to real world. For example. even though on paper IIS web server is technically amazing, In the real world, no one really uses IIS as part of their web stack. (For many different reasons). Again, all I see is arrogance. Its never SystemDs heavy handed approach thats wrong, its always everyone else's fault for their implementation issues.