4 ms·

Some of the hardest and memorable troubleshooting DNS problems (for me) are: 1. Default-DENY firewall policy mandates the admin being a DNS protocol expert an

Some of the hardest and memorable troubleshooting DNS problems (for me) are: 1. Default-DENY firewall policy mandates the admin being a DNS protocol expert and often requires a refresher (of own volition or ego-checkingj. 2. Gateway Firewall blocked the incoming response UDP of authoritative record transfers of secondary authoritative DNS. Firewall admin goofs often and delay discovery of DNS outage is often the result due to poor network error logging 3. Bastion (one kind of the split horizons) DNS server over multiview DNS is ALMOST always preferable in security theatre. 4. Forgetting to disable Firefox’s DoH after full DNS block at border gateway. Used Firefox policy for corporate and HomeLab network (during cutover from public resolvers to internal homeLab/corporate resolver). 5. Private DNSSEC root servers, setup of (in case of root DNS outages) 6. Negative cache resolution by TTL, balancing the 7. cache spillage prevention of internal corporate DNS records being exposed 8. Mastering resolv.conf (especially against incoming replacement by systemd-resolved) Most still hit me 10 years later despite re-reviewing my private DNS HOWTOs. Some of above experiences that I have posted on my website by DNS topic [1]. Website caveat: Still not sorry that Google Chrome still cannot HTTPS-negotiate for HTTP/1.3-only (ignore HTTP/2) with just only the ChaCha (no AES/RSA) algorithm. Use a different browser, that’s my firm security stance. (Most corporate firewall should be blocking HTTP/2 until their transparent HTTP proxy have been upgrade to handle HTTP/2 because its inline selective blocking within a HTTP/2 TLS stream is still a thorny hurdle.) [1] https://egbert.net/blog/tags/dns.html https://egbert.net/blog/tags/dns.html

doubled112 3y ago
That fact applications are now ignoring my DNS configuration for their own makes me irrationally angry. Android, Chrome, Firefox, probably more I haven’t noticed yet. I am the network admin here. Please follow my rules. At least now I know to check.
- throw0101a 3y ago
  > I am the network admin here. Please follow my rules. When DNS over HTTPS (DoH) was announced many of its proponents seem to dismiss, or at least downplay, this concern often.
  - egberts1 3y ago
    That's why these same DoH (and VPN) proponents are not a network security expert. They should be using a client-side-TLS-signed (and verified) DoT only at their own DNS-forward-blocking border gateway with their wireguarded remote DNS resolver unless multiple DNS views are so desired (such as QubeOS desktops) then it is down the rabbit hole for DNS experts only.
    - egberts1 3y ago
      and your firewall is default/deny. (I frequently forget to say this often but always configure for one at the start.) Corporate Bonus if you can scrub all TLS traffic at kernel level while running a transparent HTTPS/TLS proxy at the border gateway: that is, force all TLS through that proxy by payload detection mechanism and not just by port numbers.
- xaduha 3y ago
  > I am the network admin here. Please follow my rules. In an organisation - sure. But sometimes countries act like a giant man-in-the-middle redirecting/blocking forbidden sites e.g. Russia. Most popular DNS servers like 1.1.1.1 and 8.8.8.8 are forced to comply.
  - egberts1 3y ago
    That's what DoT is for, not DoH. (Looking at you, Great Firewall.)
    - xaduha 3y ago
      Neither will work by themselves as far as censorship goes because it's not just DNS, but either can solve some minor geofencing annoyances like Qt blocking access to https://download.qt.io https://download.qt.io or Spotify pointing their podcast CDN to 127.0.0.1 when using major DNS providers.
      - egberts1 3y ago
        Yea, geofencing on IP or by AS are often done. Least you can get the DNS resolver working.
- justsomehnguy 3y ago
  > probably more I haven’t noticed yet. >> May 06 16:34:03 host.contoso.com dockerd[1496]: time="2023-05-06T16:34:03.666703897Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]" >> May 06 16:34:03 host.contoso.com dockerd[1496]: time="2023-05-06T16:34:03.666734656Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]" I'm running Unbound on these machines.
  - egberts1 3y ago
    Gotta love the struggle of Docker daemon and resolver. systemd-resolved still isn't helping much for me at this either. Go to the biggest flowchart of resolv.conf in this detailed writeup. https://tailscale.com/blog/sisyphean-dns-client-linux/ https://tailscale.com/blog/sisyphean-dns-client-linux/
- phicoh 3y ago
  You are the network admin, just block access to port 443 and the problem is solved. The power of a network admin over a host admin is limited to what can be blocked. And the more that is blocked, the more traffic will move to port 443, maybe using some relays at big cloud providers. I'd say, the job a network admin is to provide positive services: reliable transport of bits, reliable DNS resolvers, etc. Beyond that, a network admin should not look at what users are doing with those bits.
  - doubled112 3y ago
    I do want my bits to transported reliably, but when I have internal services, I want my bits to stay realiably internal. Split DNS is the easiest way to achieve this, but it breaks. DoH is not the problem. I'm using it too. It is the "surprise" that my devices are not using the one given to them.
  - throw0101b 3y ago
    > I'd say, the job a network admin is to provide positive services: reliable transport of bits, reliable DNS resolvers, etc. Beyond that, a network admin should not look at what users are doing with those bits. I'll be sure to tell that to the regulator overseeing my industry and my Compliance department.
    - phicoh 3y ago
      Maybe you don't want to place that solely on the shoulders of the network admin. These days most network data is encrypted, so the network admin cannot do much. So either the host must be trusted to conform to policy. In which case it up to the host admin to avoid bad setups, or the host should have no (direct) access to the internet.
topherisswell 3y ago
Re: #1 You don't have to be DNS expert to know the ports/protocols that various DNS traffic uses. It's literally a question on every CCNA, CEH, and A+ certification exam.
- egberts1 3y ago
  Haven't found a certification for default-deny firewall. there are certifications and then there are experts.
kevin_nisbet 3y ago
Here's my favorite DNS story: https://wdkwwdk.com/posts/outage-stories-dns/ https://wdkwwdk.com/posts/outage-stories-dns/ The TLDR is in most cases DNS records with the same name but different record type are expired independent of each other. In the DNS protocol there are two different ways to respond that an answer doesn't exist, with subtle differences. We had a typo in our DNS configuration, that caused No Such Name responses on IPv6 queries... and we weren't using IPv6. No such name means the name doesn't exist for any record type, not just the record you asked for doesn't exist. This caused our resolver to look at it's cache and throw out every DNS record with that name... including ones we had statically configured. We no longer knew how to contact root, so we could no longer do any external DNS queries that needed our root servers.

doubled112 3y ago

That fact applications are now ignoring my DNS configuration for their own makes me irrationally angry. Android, Chrome, Firefox, probably more I haven’t noticed yet. I am the network admin here. Please follow my rules. At least now I know to check.

throw0101a 3y ago
> I am the network admin here. Please follow my rules. When DNS over HTTPS (DoH) was announced many of its proponents seem to dismiss, or at least downplay, this concern often.
- egberts1 3y ago
  That's why these same DoH (and VPN) proponents are not a network security expert. They should be using a client-side-TLS-signed (and verified) DoT only at their own DNS-forward-blocking border gateway with their wireguarded remote DNS resolver unless multiple DNS views are so desired (such as QubeOS desktops) then it is down the rabbit hole for DNS experts only.
  - egberts1 3y ago
    and your firewall is default/deny. (I frequently forget to say this often but always configure for one at the start.) Corporate Bonus if you can scrub all TLS traffic at kernel level while running a transparent HTTPS/TLS proxy at the border gateway: that is, force all TLS through that proxy by payload detection mechanism and not just by port numbers.
xaduha 3y ago
> I am the network admin here. Please follow my rules. In an organisation - sure. But sometimes countries act like a giant man-in-the-middle redirecting/blocking forbidden sites e.g. Russia. Most popular DNS servers like 1.1.1.1 and 8.8.8.8 are forced to comply.
- egberts1 3y ago
  That's what DoT is for, not DoH. (Looking at you, Great Firewall.)
  - xaduha 3y ago
    Neither will work by themselves as far as censorship goes because it's not just DNS, but either can solve some minor geofencing annoyances like Qt blocking access to https://download.qt.io https://download.qt.io or Spotify pointing their podcast CDN to 127.0.0.1 when using major DNS providers.
    - egberts1 3y ago
      Yea, geofencing on IP or by AS are often done. Least you can get the DNS resolver working.
justsomehnguy 3y ago
> probably more I haven’t noticed yet. >> May 06 16:34:03 host.contoso.com dockerd[1496]: time="2023-05-06T16:34:03.666703897Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]" >> May 06 16:34:03 host.contoso.com dockerd[1496]: time="2023-05-06T16:34:03.666734656Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]" I'm running Unbound on these machines.
- egberts1 3y ago
  Gotta love the struggle of Docker daemon and resolver. systemd-resolved still isn't helping much for me at this either. Go to the biggest flowchart of resolv.conf in this detailed writeup. https://tailscale.com/blog/sisyphean-dns-client-linux/ https://tailscale.com/blog/sisyphean-dns-client-linux/
phicoh 3y ago
You are the network admin, just block access to port 443 and the problem is solved. The power of a network admin over a host admin is limited to what can be blocked. And the more that is blocked, the more traffic will move to port 443, maybe using some relays at big cloud providers. I'd say, the job a network admin is to provide positive services: reliable transport of bits, reliable DNS resolvers, etc. Beyond that, a network admin should not look at what users are doing with those bits.
- doubled112 3y ago
  I do want my bits to transported reliably, but when I have internal services, I want my bits to stay realiably internal. Split DNS is the easiest way to achieve this, but it breaks. DoH is not the problem. I'm using it too. It is the "surprise" that my devices are not using the one given to them.
- throw0101b 3y ago
  > I'd say, the job a network admin is to provide positive services: reliable transport of bits, reliable DNS resolvers, etc. Beyond that, a network admin should not look at what users are doing with those bits. I'll be sure to tell that to the regulator overseeing my industry and my Compliance department.
  - phicoh 3y ago
    Maybe you don't want to place that solely on the shoulders of the network admin. These days most network data is encrypted, so the network admin cannot do much. So either the host must be trusted to conform to policy. In which case it up to the host admin to avoid bad setups, or the host should have no (direct) access to the internet.

topherisswell 3y ago

Re: #1 You don't have to be DNS expert to know the ports/protocols that various DNS traffic uses. It's literally a question on every CCNA, CEH, and A+ certification exam.

egberts1 3y ago
Haven't found a certification for default-deny firewall. there are certifications and then there are experts.

kevin_nisbet 3y ago

Here's my favorite DNS story: https://wdkwwdk.com/posts/outage-stories-dns/ https://wdkwwdk.com/posts/outage-stories-dns/ The TLDR is in most cases DNS records with the same name but different record type are expired independent of each other. In the DNS protocol there are two different ways to respond that an answer doesn't exist, with subtle differences. We had a typo in our DNS configuration, that caused No Such Name responses on IPv6 queries... and we weren't using IPv6. No such name means the name doesn't exist for any record type, not just the record you asked for doesn't exist. This caused our resolver to look at it's cache and throw out every DNS record with that name... including ones we had statically configured. We no longer knew how to contact root, so we could no longer do any external DNS queries that needed our root servers.