9 ms·

The Sisyphean Task of DNS Client Config on Linux

167 points by smitop 5y ago

bonzini 5y ago
The most interesting takeaway from this article is that, according to people who actually do the work, NetworkManager and systemd-resolved do get things right.
- liveoneggs 5y ago
  one reason resolv.conf is/was so sticky is that gethostbyname (and friends) is a libc thing and pretty low-level
  - corty 5y ago
    Problem is, modern browsers do things differently yet again. So the insanity continues, just on different levels.
    - liveoneggs 5y ago
      browsers are operating systems in user space. They will continue to replace kernel parts.
      - vetinari 5y ago
        Technically, the existing dns resolving mechanism is a part of glibc, so it is userspace already.
- YarickR2 5y ago
  Mostly right. Article never mentions /etc/hosts , which is still largely a thing, and works wonders in difficult cases ( and makes other difficult cases much worse to debug)
  - vetinari 5y ago
    Because /etc/hosts is not really part of the chain handled by resolver or nss-dns. It is handled by different nss module which usually has higher priority.
- kevinoid 5y ago
  I agree! As a non-expert, I've also had good luck with NetworkManager, but wouldn't recommend systemd-resolved yet if you use DNSSEC: https://github.com/systemd/systemd/issues/6490 https://github.com/systemd/systemd/issues/6490 (fixed in v248) https://github.com/systemd/systemd/issues/8451 https://github.com/systemd/systemd/issues/8451 https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/12388 https://github.com/systemd/systemd/issues/12388
schmichael 5y ago
Where does /etc/nsswitch.conf factor in? I don't see it mentioned.
- daenney 5y ago
  It’s probably omitted as nsswitch doesn’t affect DNS client configuration. It affects which sources are consulted to lookup a name (including maybe asking DNS), but it doesn’t configure things like which DNS servers to ask or what options to set. > The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf, is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order.
  - LukeShu 5y ago
    nsswitch.conf is what tells it whether to use the traditional nss_dns (the thing that looks at /etc/resolv.conf), or whether to use the newer nss_resolve (the thing that talks to systemd-resolved). So yeah, if the article is contrasting resolv.conf vs systemd-resolved, nsswitch.conf is how you select which of those approaches is used.
    - daenney 5y ago
      The article focussed on how /etc/resolv.conf is being managed and populated. That’s not affected by nsswitch. I only addressed why nsswitch was likely not mentioned, not that it doesn’t serve a purpose.
    - vetinari 5y ago
      Even with nss_resolve, you still have to have /etc/resolv.conf correctly populated, because some apps will ignore gethostbyname() and just parse resolv.conf and do DNS by themselves. For example, golang stdlib does that (but to the credit of golang runtime, it checks whether nsswitch.conf is in expected state and falls back to glibc if it is not). Some apps go even further and do their DNS entirely on their own (Firefox DoH controversy). nsswitch however doesn't just configure mechanism for public resolver, it is a hook where alternate mechanisms like nss-mdns or nss-mymachines can hook up.
jiqiren 5y ago
If you are just trying to use curl or some other straightforward small application - this is enough. But running Kubernetes, docker, or some other container system? This mess is much more deep... For example: there are known issues with systemd-resolved & Kubernetes (at least on Ubuntu that defaults to systemd-resolved) https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ https://kubernetes.io/docs/tasks/administer-cluster/dns-debu...
- denysvitali 5y ago
  FYI, if you don't bother opening the link, systemd-resolv sets the /etc/resolv.conf to have only 127.0.0.1 as a DNS server. You can see how things get bad when the CoreDNS pod tries to get the "upstream" DNS servers from the host /etc/resolv.conf
  - kelnos 5y ago
    I guess it shouldn't be doing that? I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf. As much as I'm not always comfortable with the larger complexity, the CoreDNS pod really needs to be doing something like the flowchart described in the article. If systemd-resolved or NetworkManager are in play, it should be using D-Bus to talk to them to get the information it needs. resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here.
    - vetinari 5y ago
      > I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf. Dnsmasq DNS support is redundant when you are running systemd-resolved. It can do everything dnsmasq does and more. > resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here. That's exactly what systemd-resolved is.
- 5y ago
corndoge 5y ago
I recommend the series "Anatomy of a Linux DNS lookup"[0]. Everything you never wanted to know. https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/ https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...
- tyingq 5y ago
  That appears to have been written prior to systemd-resolved becoming enabled by default on many distros.
  - egberts1 5y ago
    Which systemd is still unable to do DNS split horizon with DNSSEC. - ISC Bind9 is still king.
tptacek 5y ago
Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.internal" and "rabbitmq-cluster.internal". What you'd want in a perfect work is an option in `/etc/resolv.conf` that sends `.internal` to a special nameserver, and everything else to a normal nameserver. But as far as I can tell, there's no way to take a bare Linux VM that can accept an arbitrary container and set that capability up. So instead, we end up (by default; you could override) serving _all_ customer DNS, and our `.internal` server has to forward recursive queries to things that aren't `.internal` somewhere else. This sucks; we shouldn't have to be inline for arbitrary customer DNS. If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.
- friseurtermin 5y ago
  Funnily enough, that is the _exact_ same problem I'm facing right now, down to Firecracker and a custom internal TLD. I'm excited to see the solutions. I think the only difference is that I need to run this DNS service on the same host as my VMs, so I will need to use a different port than systemd-resolve.
  - YarickR2 5y ago
    I'd recommend to run it on the same port, but different IP (127.0.1.2 , f.e.) , due to inability of some programs to use non-standard port
  - friseurtermin 5y ago
    So apparently systemd-resolve does support redirecting custom TLDs to a specific nameserver [0], as mentioned by dnr. However, support for custom ports was only added in version 246 and my Ubuntu 18.04 ships with 237. As far as I can tell, my way forward is to use v246, then set my TLD to my custom nameserver with my custom port and add a stub listener on localhost (instead of 127.0.0.53, which is the default). Then I can offer my VMs to use their tap gateway (which is the Ubuntu host) as a DNS server if they want, and that will resolve my custom TLD and forward all other stuff to its own nameserver. Or, if the user wants, they can also do their own stuff. It seems that a lot has happened with systemd-resolve in the last few months.
denysvitali 5y ago
Ok, now let's talk about how messed up the same thing is in Mac OS. Hint: it is waaay worse
- xrd 5y ago
  Dave, one of the authors of these posts, has been regularly documenting on Twitter his struggles with DNS on all the platforms. If you like sports, it is like watching a basketball game, with three teams and Dave as the play by play announcer. I'm a linux user, so I'm always cheering when OSX or Windows "loses" but then very quickly Dave will, paraphrasing say: "And, OSX hits a deep 3 pointer..." ("And, OSX does this right, and Linux sucks"). But, it is great theater and very informative. On Dave's recommendation, I switched to systemd-resolved, and it is working flawlessly for me (with my own wireguard setup, not using Tailscale yet).
- dave_universetf 5y ago
  It's a pity, because macOS got the general idea right, but seemingly every single particular wrong thereafter. In general, a modern DNS client wants: a set of "default route" resolvers; a set of "DNS routes" that point certain suffixes to other resolver configs; a set of search paths to expand single-label queries; integration with mdns and LLMNR, for seamless zero-config resolution on LANs (super important for printers, in particular); all of the above tied to interface lifetimes, so you can tie resolver reachability to underlying network state; very detailed documentation on the algorithm used to resolve a name, and how you traverse all the above configuration. macOS has default resolvers, DNS routes, mdns integration (but no LLMNR), interface-tied configs, and knows about search paths. But then you look at the NetworkExtension API for configuring DNS, and it turns out the search paths field doesn't actually configure the search paths in ways you'd expect, instead all the suffixes you install as "routes" end up also becoming search paths, and your only option is have all or none of them be search paths. Meanwhile, the search paths you specified do get installed... In an interface-scoped config that doesn't actually get used in the majority of name lookups that need name expansion. It's so frustrating because it's this close to being excellent, and instead ends up being the most limiting of APIs we have to work with, because being apple, it's either their API or go screw yourself and don't configure DNS. Oh dear, I've ranted again, haven't I. Anyway, every OS is its own beautiful little snowflake of weirdery an brokenness. Linux's particular flavor is "there's 15 ways to do it, most of which require polyfills". macOS's flavor is "we have an API that should be amazing but somehow does the wrong thing almost always". Windows's flavor is "we can do really cool things but the main source of documentation is people exchanging superstitions about registry keys on stack overflow". Given that choice, I think I prefer linux. It's way more code to write to make it work, but at least the code can be derived from documentation+source code, and has half a chance of working as desired.
thdxr 5y ago
I've been avoiding figuring this out years and resorting to editing /etc/resolv.conf manually and chattr +i it to make it readonly. I'm a NetworkManager user so guess it's time to set it up through there
venamresm__ 5y ago
I also tried once to make sense of domain name resolution: https://venam.nixers.net/blog/unix/2020/11/01/resolving-a-hostname.html https://venam.nixers.net/blog/unix/2020/11/01/resolving-a-ho...
Koffiepoeder 5y ago
To be honest I hate all of the aforementioned programs trying to battle over /etc/resvolv.conf. That's why by default I have the file marked as immutable, and pointing to 127.0.0.1. This way I cannot have accidental DNS traffic leaking. If I need local network DNS I send a manual dhcp command to get the wifi's DNS and add it temporarily to my dnscrypt config. Similar thing for VPN DNS.
- mindslight 5y ago
  I do something similar whereby I just point DNS traffic at the router itself and configure it per-host by DNAT, as part of the larger WAN horizon zone (eg direct goes to local recursor, wireguard tunnel goes to the recursor on the other side of the tunnel or suitable public resolver, commercial VPN goes to whatever their recommended resolver is, eventually pihole for guest network). Running multiple apps / security contexts / nyms (or whatever else you want to call them) on the same OS instance is just asking for trouble.
bkus 5y ago
But wait, there's more! nsswitch.conf and nscd.conf also affect the stub resolver's behavior.
tgbugs 5y ago
An enlightening article that explains why I think I am losing my mind every time I try to get the network configured correctly on any number of distros. My main take home is to never install NetworkManager or a resolvconf and to avoid systemd if at all possible, unless I absolutely know that I am going to need those state of the art DNS capabilities. I have mostly managed to avoid resolv.conf issues since the default Gentoo image has a sane setup (even if using dhcp on a laptop and switching wifi and wired). However, every single time I have tried to set up a system using another distro something has gone wrong, and in support of the theory that NetworkManager is a major contributor to the problem the one Gentoo system with NetworkManager installed had the same issues. To echo the plea in the original article, in nearly every case the primary challenge has been to figure out exactly what documentation actually applies to the system at hand because distros seem to change this completely out of sync with any attempt to correct or align the documentation. I suspect that on an individual user level this leads to a happy path situation where everyone who does the right thing by accident is quiet and the ones who need something slightly different are never heard from because they weren't able to even connect to the internet (probably not quiet that bad).
- vetinari 5y ago
  That's exactly the wrong take home; the only sane way to handle dns is systemd-resolved, as the article said. If you use your machine as anything resebling desktop/laptop, you should configure your network using Network Manager, especially if you have connections that come and go. Switching between Lan and wifi with dhcp is really low bar. Anything else is just prolonging the agony that the linux networking configuration had to endure for years.
koprulusector 5y ago
Wait wait wait. You forgot the part where if using systemd-resolved you are supposed to symlink /etc/resolv.conf to /run/systemd/resolve/resolve.conf, which gets generated after boot. Also should caveat that you will no longer be able to resolve / recursive search based on only a hostname, services using systemd-resolved instead of glibc will need to provide fqdn. Real pain in the ass.
skywhopper 5y ago
I’m confused, this article seems to assert that systemd-resolved can handle split DNS based on the domain name and even claims the docs for it are fantastic. But last time I went down this rabbit hole I finally gave up and combined it with dnsmasq. The docs are verbose but useless, and I ultimately determined it was impossible. systemd-resolved would failover to alternate DNS servers but you’d end up making a ton of failed requests. So if I’m wrong and you can actually send requests to DNS servers based on their TLD please tell me how.
- cp9 5y ago
  It depends on how long ago you checked, but it can do it now without dnsmasq. https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b567 https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...
newman314 5y ago
Wait till you see what happens when Zscaler gets added into the mix for corporate laptops. I now have to scenario for when Zscaler intercepts/rewrites DNS requests along with when the Cisco VPN is active. Given that a large amount of folks are still WFH, it’s enough to make me want to tear the remainder of my hair out.
- haik90 5y ago
  Can't agree more. They have 'clever' way doing a lot of things, which make us suffering to troubleshooting
egberts1 5y ago
systemd-resolved still doesn’t work with split-horizon DNS with DNSSEC on both sides of the horizon.

167 points by smitop 5y ago

The most interesting takeaway from this article is that, according to people who actually do the work, NetworkManager and systemd-resolved do get things right.

liveoneggs 5y ago
one reason resolv.conf is/was so sticky is that gethostbyname (and friends) is a libc thing and pretty low-level
- corty 5y ago
  Problem is, modern browsers do things differently yet again. So the insanity continues, just on different levels.
  - liveoneggs 5y ago
    browsers are operating systems in user space. They will continue to replace kernel parts.
    - vetinari 5y ago
      Technically, the existing dns resolving mechanism is a part of glibc, so it is userspace already.
YarickR2 5y ago
Mostly right. Article never mentions /etc/hosts , which is still largely a thing, and works wonders in difficult cases ( and makes other difficult cases much worse to debug)
- vetinari 5y ago
  Because /etc/hosts is not really part of the chain handled by resolver or nss-dns. It is handled by different nss module which usually has higher priority.
kevinoid 5y ago
I agree! As a non-expert, I've also had good luck with NetworkManager, but wouldn't recommend systemd-resolved yet if you use DNSSEC: https://github.com/systemd/systemd/issues/6490 https://github.com/systemd/systemd/issues/6490 (fixed in v248) https://github.com/systemd/systemd/issues/8451 https://github.com/systemd/systemd/issues/8451 https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/12388 https://github.com/systemd/systemd/issues/12388

schmichael 5y ago

Where does /etc/nsswitch.conf factor in? I don't see it mentioned.

daenney 5y ago
It’s probably omitted as nsswitch doesn’t affect DNS client configuration. It affects which sources are consulted to lookup a name (including maybe asking DNS), but it doesn’t configure things like which DNS servers to ask or what options to set. > The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf, is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order.
- LukeShu 5y ago
  nsswitch.conf is what tells it whether to use the traditional nss_dns (the thing that looks at /etc/resolv.conf), or whether to use the newer nss_resolve (the thing that talks to systemd-resolved). So yeah, if the article is contrasting resolv.conf vs systemd-resolved, nsswitch.conf is how you select which of those approaches is used.
  - daenney 5y ago
    The article focussed on how /etc/resolv.conf is being managed and populated. That’s not affected by nsswitch. I only addressed why nsswitch was likely not mentioned, not that it doesn’t serve a purpose.
  - vetinari 5y ago
    Even with nss_resolve, you still have to have /etc/resolv.conf correctly populated, because some apps will ignore gethostbyname() and just parse resolv.conf and do DNS by themselves. For example, golang stdlib does that (but to the credit of golang runtime, it checks whether nsswitch.conf is in expected state and falls back to glibc if it is not). Some apps go even further and do their DNS entirely on their own (Firefox DoH controversy). nsswitch however doesn't just configure mechanism for public resolver, it is a hook where alternate mechanisms like nss-mdns or nss-mymachines can hook up.

jiqiren 5y ago

If you are just trying to use curl or some other straightforward small application - this is enough. But running Kubernetes, docker, or some other container system? This mess is much more deep... For example: there are known issues with systemd-resolved & Kubernetes (at least on Ubuntu that defaults to systemd-resolved) https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ https://kubernetes.io/docs/tasks/administer-cluster/dns-debu...

denysvitali 5y ago
FYI, if you don't bother opening the link, systemd-resolv sets the /etc/resolv.conf to have only 127.0.0.1 as a DNS server. You can see how things get bad when the CoreDNS pod tries to get the "upstream" DNS servers from the host /etc/resolv.conf
- kelnos 5y ago
  I guess it shouldn't be doing that? I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf. As much as I'm not always comfortable with the larger complexity, the CoreDNS pod really needs to be doing something like the flowchart described in the article. If systemd-resolved or NetworkManager are in play, it should be using D-Bus to talk to them to get the information it needs. resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here.
  - vetinari 5y ago
    > I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf. Dnsmasq DNS support is redundant when you are running systemd-resolved. It can do everything dnsmasq does and more. > resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here. That's exactly what systemd-resolved is.
5y ago

corndoge 5y ago

I recommend the series "Anatomy of a Linux DNS lookup"[0]. Everything you never wanted to know. https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/ https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...

tyingq 5y ago
That appears to have been written prior to systemd-resolved becoming enabled by default on many distros.
- egberts1 5y ago
  Which systemd is still unable to do DNS split horizon with DNSSEC. - ISC Bind9 is still king.

tptacek 5y ago

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.internal" and "rabbitmq-cluster.internal". What you'd want in a perfect work is an option in `/etc/resolv.conf` that sends `.internal` to a special nameserver, and everything else to a normal nameserver. But as far as I can tell, there's no way to take a bare Linux VM that can accept an arbitrary container and set that capability up. So instead, we end up (by default; you could override) serving _all_ customer DNS, and our `.internal` server has to forward recursive queries to things that aren't `.internal` somewhere else. This sucks; we shouldn't have to be inline for arbitrary customer DNS. If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.

friseurtermin 5y ago
Funnily enough, that is the _exact_ same problem I'm facing right now, down to Firecracker and a custom internal TLD. I'm excited to see the solutions. I think the only difference is that I need to run this DNS service on the same host as my VMs, so I will need to use a different port than systemd-resolve.
- YarickR2 5y ago
  I'd recommend to run it on the same port, but different IP (127.0.1.2 , f.e.) , due to inability of some programs to use non-standard port
- friseurtermin 5y ago
  So apparently systemd-resolve does support redirecting custom TLDs to a specific nameserver [0], as mentioned by dnr. However, support for custom ports was only added in version 246 and my Ubuntu 18.04 ships with 237. As far as I can tell, my way forward is to use v246, then set my TLD to my custom nameserver with my custom port and add a stub listener on localhost (instead of 127.0.0.53, which is the default). Then I can offer my VMs to use their tap gateway (which is the Ubuntu host) as a DNS server if they want, and that will resolve my custom TLD and forward all other stuff to its own nameserver. Or, if the user wants, they can also do their own stuff. It seems that a lot has happened with systemd-resolve in the last few months.

denysvitali 5y ago

Ok, now let's talk about how messed up the same thing is in Mac OS. Hint: it is waaay worse

xrd 5y ago
Dave, one of the authors of these posts, has been regularly documenting on Twitter his struggles with DNS on all the platforms. If you like sports, it is like watching a basketball game, with three teams and Dave as the play by play announcer. I'm a linux user, so I'm always cheering when OSX or Windows "loses" but then very quickly Dave will, paraphrasing say: "And, OSX hits a deep 3 pointer..." ("And, OSX does this right, and Linux sucks"). But, it is great theater and very informative. On Dave's recommendation, I switched to systemd-resolved, and it is working flawlessly for me (with my own wireguard setup, not using Tailscale yet).
dave_universetf 5y ago
It's a pity, because macOS got the general idea right, but seemingly every single particular wrong thereafter. In general, a modern DNS client wants: a set of "default route" resolvers; a set of "DNS routes" that point certain suffixes to other resolver configs; a set of search paths to expand single-label queries; integration with mdns and LLMNR, for seamless zero-config resolution on LANs (super important for printers, in particular); all of the above tied to interface lifetimes, so you can tie resolver reachability to underlying network state; very detailed documentation on the algorithm used to resolve a name, and how you traverse all the above configuration. macOS has default resolvers, DNS routes, mdns integration (but no LLMNR), interface-tied configs, and knows about search paths. But then you look at the NetworkExtension API for configuring DNS, and it turns out the search paths field doesn't actually configure the search paths in ways you'd expect, instead all the suffixes you install as "routes" end up also becoming search paths, and your only option is have all or none of them be search paths. Meanwhile, the search paths you specified do get installed... In an interface-scoped config that doesn't actually get used in the majority of name lookups that need name expansion. It's so frustrating because it's this close to being excellent, and instead ends up being the most limiting of APIs we have to work with, because being apple, it's either their API or go screw yourself and don't configure DNS. Oh dear, I've ranted again, haven't I. Anyway, every OS is its own beautiful little snowflake of weirdery an brokenness. Linux's particular flavor is "there's 15 ways to do it, most of which require polyfills". macOS's flavor is "we have an API that should be amazing but somehow does the wrong thing almost always". Windows's flavor is "we can do really cool things but the main source of documentation is people exchanging superstitions about registry keys on stack overflow". Given that choice, I think I prefer linux. It's way more code to write to make it work, but at least the code can be derived from documentation+source code, and has half a chance of working as desired.

thdxr 5y ago

I've been avoiding figuring this out years and resorting to editing /etc/resolv.conf manually and chattr +i it to make it readonly. I'm a NetworkManager user so guess it's time to set it up through there

venamresm__ 5y ago

I also tried once to make sense of domain name resolution: https://venam.nixers.net/blog/unix/2020/11/01/resolving-a-hostname.html https://venam.nixers.net/blog/unix/2020/11/01/resolving-a-ho...

Koffiepoeder 5y ago

To be honest I hate all of the aforementioned programs trying to battle over /etc/resvolv.conf. That's why by default I have the file marked as immutable, and pointing to 127.0.0.1. This way I cannot have accidental DNS traffic leaking. If I need local network DNS I send a manual dhcp command to get the wifi's DNS and add it temporarily to my dnscrypt config. Similar thing for VPN DNS.

mindslight 5y ago
I do something similar whereby I just point DNS traffic at the router itself and configure it per-host by DNAT, as part of the larger WAN horizon zone (eg direct goes to local recursor, wireguard tunnel goes to the recursor on the other side of the tunnel or suitable public resolver, commercial VPN goes to whatever their recommended resolver is, eventually pihole for guest network). Running multiple apps / security contexts / nyms (or whatever else you want to call them) on the same OS instance is just asking for trouble.

bkus 5y ago

But wait, there's more! nsswitch.conf and nscd.conf also affect the stub resolver's behavior.

tgbugs 5y ago

An enlightening article that explains why I think I am losing my mind every time I try to get the network configured correctly on any number of distros. My main take home is to never install NetworkManager or a resolvconf and to avoid systemd if at all possible, unless I absolutely know that I am going to need those state of the art DNS capabilities. I have mostly managed to avoid resolv.conf issues since the default Gentoo image has a sane setup (even if using dhcp on a laptop and switching wifi and wired). However, every single time I have tried to set up a system using another distro something has gone wrong, and in support of the theory that NetworkManager is a major contributor to the problem the one Gentoo system with NetworkManager installed had the same issues. To echo the plea in the original article, in nearly every case the primary challenge has been to figure out exactly what documentation actually applies to the system at hand because distros seem to change this completely out of sync with any attempt to correct or align the documentation. I suspect that on an individual user level this leads to a happy path situation where everyone who does the right thing by accident is quiet and the ones who need something slightly different are never heard from because they weren't able to even connect to the internet (probably not quiet that bad).

vetinari 5y ago
That's exactly the wrong take home; the only sane way to handle dns is systemd-resolved, as the article said. If you use your machine as anything resebling desktop/laptop, you should configure your network using Network Manager, especially if you have connections that come and go. Switching between Lan and wifi with dhcp is really low bar. Anything else is just prolonging the agony that the linux networking configuration had to endure for years.

koprulusector 5y ago

Wait wait wait. You forgot the part where if using systemd-resolved you are supposed to symlink /etc/resolv.conf to /run/systemd/resolve/resolve.conf, which gets generated after boot. Also should caveat that you will no longer be able to resolve / recursive search based on only a hostname, services using systemd-resolved instead of glibc will need to provide fqdn. Real pain in the ass.

skywhopper 5y ago

I’m confused, this article seems to assert that systemd-resolved can handle split DNS based on the domain name and even claims the docs for it are fantastic. But last time I went down this rabbit hole I finally gave up and combined it with dnsmasq. The docs are verbose but useless, and I ultimately determined it was impossible. systemd-resolved would failover to alternate DNS servers but you’d end up making a ton of failed requests. So if I’m wrong and you can actually send requests to DNS servers based on their TLD please tell me how.

cp9 5y ago
It depends on how long ago you checked, but it can do it now without dnsmasq. https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b567 https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...

newman314 5y ago

Wait till you see what happens when Zscaler gets added into the mix for corporate laptops. I now have to scenario for when Zscaler intercepts/rewrites DNS requests along with when the Cisco VPN is active. Given that a large amount of folks are still WFH, it’s enough to make me want to tear the remainder of my hair out.

haik90 5y ago
Can't agree more. They have 'clever' way doing a lot of things, which make us suffering to troubleshooting

egberts1 5y ago

systemd-resolved still doesn’t work with split-horizon DNS with DNSSEC on both sides of the horizon.