7 ms·

Some ways DNS can break

106 points by vinnyglennon 5y ago

treffer 5y ago
Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there) Someone has configured a delegation loop. (Yep, been there more than once) Records are currently unavailable, but someone has busted the cache. (Enjoy) All services wait for the same DNS record to be fetched and will all ask the same service at exactly the time when DNS is done. (Again, clear yes, happens. That's why you should use prefetch) No one knows how to handle DNS SRV priority correctly (please correct me if I am wrong).
- richardwhiuk 5y ago
  The problem with DNS SRV priority is that it rarely provides the behaviour you actually want.
- tinus_hn 5y ago
  I remember when you could list entire cctlds, how did that work then? They weren’t as big as they are today but definitely bigger than 65k.
  - matttpt 5y ago
    It sounds like you're thinking of zone transfers (the special AXFR query type). In that case the server's response may consist of more than one DNS message sent consecutively over a single TCP connection [1], so it's possible to transfer large zones. E.g. out of curiosity I just asked for the root zone with dig . axfr @iad.xfr.dns.icann.org and I received 24,448 records transferred over 83 DNS messages, according to the stats dig prints at the end. [1] https://datatracker.ietf.org/doc/html/rfc5936#section-2.2 https://datatracker.ietf.org/doc/html/rfc5936#section-2.2
- bogomipz 5y ago
  >Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there) I am curious how did you get there given the length field in DNS records is only 16 bits?
  - grogers 5y ago
    I think the point was that they wanted to return more data than 64kB in the response but couldn't.
- bradknowles 5y ago
  SRV is deprecated. Don’t use it at all.
  - oogali 5y ago
    When did SRV become deprecated? The standard[1] doesn’t show it was obsoleted by anything else. 1: https://datatracker.ietf.org/doc/html/rfc2782 https://datatracker.ietf.org/doc/html/rfc2782
    - bradknowles 5y ago
      I think it was one of the BCPs that recommended that people not use SRV records. I’ll have to check.
kiallmacinnes 5y ago
On the `ndots` problem, it's best to get in the habit of using fully qualified names when you know them. A fully qualified DNS name ends in a period. Continuing the example in the article, use `google.com.` and there will be just one query.
- AndyMcConachie 5y ago
  You can kinda fix this by lowering the ndots value. But probably the better way to fix this is to not include a 'search' directive at all. Don't specify anything for 'search' or 'ndots'. And then do as you say, use FQDNs everywhere.
  - kiallmacinnes 5y ago
    That's not always possible & wise, e.g. the example given is the Kubernetes search/ndots values. Changing this would break the expected "API" between the applications in Kube and the Kube cluster, which is bound to cause some pain and suffering down the line :) Using a FQDN (as in, a DNS name with the trailing dot) is the only reliable way to avoid this without breaking other things!
- m3047 5y ago
  This doesn't work reliably, because TLS gets implemented incorrectly and the trailing dot is not stripped from the SNI host (TLS not HTTP) header. Curl has gone back and forth; dunno whether that is curl itself or the resolver libs. Firefox used to be broken, and may still be. Apache used to almost blow chunks when it got one of these, and I filed a bug (https://bz.apache.org/bugzilla/show_bug.cgi?id=58007); https://bz.apache.org/bugzilla/show_bug.cgi?id=58007); the chosen fix was to serve some "random" cert. Just tried it with an older version of FFX which I know is borked though and it did the right thing, so presumably something further has occurred since 2017.
AndyMcConachie 5y ago
This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system? We've come a long way from good ole fashioned libc/glibc acting as a good ole fashioned stub resolver on my UNIX system. Is systemd even looking at /etc/resolv.conf anymore? Who is overwriting /etc/resolv.conf? Is the information provided via DHCP being ignored? Is Firefox using DoH? Where is caching happening? How do I clear it? All of these questions have completely different answers depending on OS and how they're configured. I work with OSX, Debian, Redhat and OpenBSD on the regular and they all have different answers for the above questions.
- kiallmacinnes 5y ago
  > This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system? This does my nut in too - even just narrowing the Q down to modern Linux Desktops with e.g. chrome/firefox, and I have absolutely no idea what the query path is. So many components have been added to the path over the last few years :(
- octoberfranklin 5y ago
  > Who is overwriting /etc/resolv.conf? echo 'nameserver 127.0.0.1' > /etc/resolv.conf chattr +i /etc/resolv.conf You can thank me later.
  - klapatsibalo 5y ago
    > A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute. So you essentially block any program trying to write it?
    - tinus_hn 5y ago
lewisl9029 5y ago
One recent frustration I ran into with Node's DNS is that it doesn't seem to resolve whatever.localhost to 127.0.0.1, unlike my browser. Not entirely sure which is doing the right thing by the spec here, but I much prefer the browser behavior, because I would really like to not have to add a hosts entry for everything I reverse proxy from Caddy for doing local HTTPS/HTTP2 development for production parity. Oh, and I also found out that Node also isn't able to use the system-wide certs store (which local HTTPS in Caddy adds to) [1], so in the end even adding a hosts entry didn't work, and I had to revert to plaintext HTTP1. Deno is looking more attractive every day [2]. Too bad I can't use it for building VSCode extensions. Really curious to hear how other folks are doing local HTTPS/HTTP2 setups in node these days. [1] https://github.com/nodejs/node/issues/3159 https://github.com/nodejs/node/issues/3159 [2] https://github.com/denoland/deno/pull/11491 https://github.com/denoland/deno/pull/11491
- withinboredom 5y ago
  Node (community?) does a lot of weird things when it comes to localhost. No idea if this is still true, but a lot of libraries/servers bound to 127.0.0.1 instead of localhost. Watching it explode in an ipv6 environment was cute. Then there was Docker and that didn’t work because the socket wasn’t bound to a public socket. I don’t know what 0.0.0.0 wasn’t the default in these libraries. Maybe it was a hint it wasn’t to be used in production…
- mhio 5y ago
  > resolve whatever.localhost to 127.0.0.1 I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup. I generally have a wildcard setup on an internal domain for generic local names (*.local.me) > Node also isn't able to use the system-wide certs store Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch. https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#node_extra_ca_certsfile https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#node_...
  - lewisl9029 5y ago
    > I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup. Fair enough, I guess I just wish node would behave more like a browser in this case then, as it'd have saved me a lot of headache. > I generally have a wildcard setup on an internal domain for generic local names (*.local.me) Curious how you set that up? Hosts files don't allow wildcards afaik, and that's what I've been struggling with. > Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch. I actually tried this option briefly but couldn't get it to work, at least not with the root cert that Caddy generated. I would much prefer a `NODE_USE_SYSTEM_CERTS` option that doesn't require pointing to any specific paths though, as there's less room for misconfiguration, which I can't rule out in my case.
peterburkimsher 5y ago
"Why do people find DNS so difficult? It’s just cache invalidation and naming things." -- Jeff Waugh https://twitter.com/jdub/status/739110670562557952?lang=en https://twitter.com/jdub/status/739110670562557952?lang=en
- Lvl999Noob 5y ago
  I read his name as "Jeff Laugh". I assume that's what he was doing when he wrote that tweet.
teddyh 5y ago
> dig ignores /etc/hosts Yes, when debugging host name resolution for a specific host, don’t use dig; use “getent hosts <name>”. This will use the normal configured host name resolution path. (Personally I use "getent ahosts <name>" to get both IPv6 and IPv4 addresses.)
- bradknowles 5y ago
  Use nslookup. The dig tool intentionally goes directly to DNS and bypasses all local overrides like NIS, /etc/hosts, etc….
  - xchaotic 5y ago
    I didn’t know about the dig behaviours but when I read about it I immediately thought that is a great way to narrow down the problem - compare dig results with something else that does respect local conf.
    - bradknowles 5y ago
      That is exactly how I always used it. If dig provided different answers than nslookup, then we knew where to start looking.
  - teddyh 5y ago
    nslookup also goes directly to DNS, as far as I can tell. From what I remember, nslookup is what people used to use before “dig” became popular. So there is no good reason to ever use nslookup.
    - bradknowles 5y ago
      It depends on how your nslookup application is configured. They can be configured to go straight to DNS and not use any local host resolution, that’s true. And in that case, I consider it to be marginally useful — at least it generates less noise than dig. However, by default on most platforms, nslookup is specifically configured to use /etc/hosts (or the equivalent) or other local lookup facilities (like NIS) before using DNS. That’s the real reason you use nslookup, to see what those other local name resolution facilities might tell you that are not represented in the DNS.
Croftengea 5y ago
Also, throw in all misconfigurations related to DNSSEC and you are in hell.
bullen 5y ago
Like last time I'll post a simple snippet of Java with dns4j that solves this weeks question: >problem: Java caching DNS records forever SimpleResolver resolver = new SimpleResolver("8.8.8.8"); Name name = new Name(host + "."); Record a = Record.newRecord(name, Type.A, DClass.IN); Message query = Message.newQuery(a); Message response = resolver.send(query); Record[] answers = response.getSectionArray(Section.ANSWER);
nisa 5y ago
problem: systemd-resolved fails randomly: https://github.com/systemd/systemd/issues/19118 https://github.com/systemd/systemd/issues/19118 - this happened to me on two arch linux installations for different domains - I was unable to debug this - would love to somehow get this fixed but I switched to dnsmasq that works fine for me.
m3047 5y ago
There is the suggestion made in the article that you can get any random IDN, and to the best of my knowledge that is not true. There's something about collating characters (there is no collation order for poop, hearts or astrological signs) only, and it has to be in only one character set (no mixed languages). That said, you are free to create "<poop>.example.com", presuming you own example.com. Indeed, "`rm -rf *`.example.com" is a valid FQDN according to the DNS (protocol) as opposed to The DNS (implementation).

106 points by vinnyglennon 5y ago

Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there) Someone has configured a delegation loop. (Yep, been there more than once) Records are currently unavailable, but someone has busted the cache. (Enjoy) All services wait for the same DNS record to be fetched and will all ask the same service at exactly the time when DNS is done. (Again, clear yes, happens. That's why you should use prefetch) No one knows how to handle DNS SRV priority correctly (please correct me if I am wrong).

richardwhiuk 5y ago
The problem with DNS SRV priority is that it rarely provides the behaviour you actually want.
tinus_hn 5y ago
I remember when you could list entire cctlds, how did that work then? They weren’t as big as they are today but definitely bigger than 65k.
- matttpt 5y ago
  It sounds like you're thinking of zone transfers (the special AXFR query type). In that case the server's response may consist of more than one DNS message sent consecutively over a single TCP connection [1], so it's possible to transfer large zones. E.g. out of curiosity I just asked for the root zone with dig . axfr @iad.xfr.dns.icann.org and I received 24,448 records transferred over 83 DNS messages, according to the stats dig prints at the end. [1] https://datatracker.ietf.org/doc/html/rfc5936#section-2.2 https://datatracker.ietf.org/doc/html/rfc5936#section-2.2
bogomipz 5y ago
>Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there) I am curious how did you get there given the length field in DNS records is only 16 bits?
- grogers 5y ago
  I think the point was that they wanted to return more data than 64kB in the response but couldn't.
bradknowles 5y ago
SRV is deprecated. Don’t use it at all.
- oogali 5y ago
  When did SRV become deprecated? The standard[1] doesn’t show it was obsoleted by anything else. 1: https://datatracker.ietf.org/doc/html/rfc2782 https://datatracker.ietf.org/doc/html/rfc2782
  - bradknowles 5y ago
    I think it was one of the BCPs that recommended that people not use SRV records. I’ll have to check.

kiallmacinnes 5y ago

On the `ndots` problem, it's best to get in the habit of using fully qualified names when you know them. A fully qualified DNS name ends in a period. Continuing the example in the article, use `google.com.` and there will be just one query.

AndyMcConachie 5y ago
You can kinda fix this by lowering the ndots value. But probably the better way to fix this is to not include a 'search' directive at all. Don't specify anything for 'search' or 'ndots'. And then do as you say, use FQDNs everywhere.
- kiallmacinnes 5y ago
  That's not always possible & wise, e.g. the example given is the Kubernetes search/ndots values. Changing this would break the expected "API" between the applications in Kube and the Kube cluster, which is bound to cause some pain and suffering down the line :) Using a FQDN (as in, a DNS name with the trailing dot) is the only reliable way to avoid this without breaking other things!
m3047 5y ago
This doesn't work reliably, because TLS gets implemented incorrectly and the trailing dot is not stripped from the SNI host (TLS not HTTP) header. Curl has gone back and forth; dunno whether that is curl itself or the resolver libs. Firefox used to be broken, and may still be. Apache used to almost blow chunks when it got one of these, and I filed a bug (https://bz.apache.org/bugzilla/show_bug.cgi?id=58007); https://bz.apache.org/bugzilla/show_bug.cgi?id=58007); the chosen fix was to serve some "random" cert. Just tried it with an older version of FFX which I know is borked though and it did the right thing, so presumably something further has occurred since 2017.

AndyMcConachie 5y ago

This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system? We've come a long way from good ole fashioned libc/glibc acting as a good ole fashioned stub resolver on my UNIX system. Is systemd even looking at /etc/resolv.conf anymore? Who is overwriting /etc/resolv.conf? Is the information provided via DHCP being ignored? Is Firefox using DoH? Where is caching happening? How do I clear it? All of these questions have completely different answers depending on OS and how they're configured. I work with OSX, Debian, Redhat and OpenBSD on the regular and they all have different answers for the above questions.

kiallmacinnes 5y ago
> This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system? This does my nut in too - even just narrowing the Q down to modern Linux Desktops with e.g. chrome/firefox, and I have absolutely no idea what the query path is. So many components have been added to the path over the last few years :(
octoberfranklin 5y ago
> Who is overwriting /etc/resolv.conf? echo 'nameserver 127.0.0.1' > /etc/resolv.conf chattr +i /etc/resolv.conf You can thank me later.
- klapatsibalo 5y ago
  > A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute. So you essentially block any program trying to write it?
  - tinus_hn 5y ago

lewisl9029 5y ago

One recent frustration I ran into with Node's DNS is that it doesn't seem to resolve whatever.localhost to 127.0.0.1, unlike my browser. Not entirely sure which is doing the right thing by the spec here, but I much prefer the browser behavior, because I would really like to not have to add a hosts entry for everything I reverse proxy from Caddy for doing local HTTPS/HTTP2 development for production parity. Oh, and I also found out that Node also isn't able to use the system-wide certs store (which local HTTPS in Caddy adds to) [1], so in the end even adding a hosts entry didn't work, and I had to revert to plaintext HTTP1. Deno is looking more attractive every day [2]. Too bad I can't use it for building VSCode extensions. Really curious to hear how other folks are doing local HTTPS/HTTP2 setups in node these days. [1] https://github.com/nodejs/node/issues/3159 https://github.com/nodejs/node/issues/3159 [2] https://github.com/denoland/deno/pull/11491 https://github.com/denoland/deno/pull/11491

withinboredom 5y ago
Node (community?) does a lot of weird things when it comes to localhost. No idea if this is still true, but a lot of libraries/servers bound to 127.0.0.1 instead of localhost. Watching it explode in an ipv6 environment was cute. Then there was Docker and that didn’t work because the socket wasn’t bound to a public socket. I don’t know what 0.0.0.0 wasn’t the default in these libraries. Maybe it was a hint it wasn’t to be used in production…
mhio 5y ago
> resolve whatever.localhost to 127.0.0.1 I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup. I generally have a wildcard setup on an internal domain for generic local names (*.local.me) > Node also isn't able to use the system-wide certs store Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch. https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#node_extra_ca_certsfile https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#node_...
- lewisl9029 5y ago
  > I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup. Fair enough, I guess I just wish node would behave more like a browser in this case then, as it'd have saved me a lot of headache. > I generally have a wildcard setup on an internal domain for generic local names (*.local.me) Curious how you set that up? Hosts files don't allow wildcards afaik, and that's what I've been struggling with. > Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch. I actually tried this option briefly but couldn't get it to work, at least not with the root cert that Caddy generated. I would much prefer a `NODE_USE_SYSTEM_CERTS` option that doesn't require pointing to any specific paths though, as there's less room for misconfiguration, which I can't rule out in my case.

peterburkimsher 5y ago

"Why do people find DNS so difficult? It’s just cache invalidation and naming things." -- Jeff Waugh https://twitter.com/jdub/status/739110670562557952?lang=en https://twitter.com/jdub/status/739110670562557952?lang=en

Lvl999Noob 5y ago
I read his name as "Jeff Laugh". I assume that's what he was doing when he wrote that tweet.

teddyh 5y ago

> dig ignores /etc/hosts Yes, when debugging host name resolution for a specific host, don’t use dig; use “getent hosts <name>”. This will use the normal configured host name resolution path. (Personally I use "getent ahosts <name>" to get both IPv6 and IPv4 addresses.)

bradknowles 5y ago
Use nslookup. The dig tool intentionally goes directly to DNS and bypasses all local overrides like NIS, /etc/hosts, etc….
- xchaotic 5y ago
  I didn’t know about the dig behaviours but when I read about it I immediately thought that is a great way to narrow down the problem - compare dig results with something else that does respect local conf.
  - bradknowles 5y ago
    That is exactly how I always used it. If dig provided different answers than nslookup, then we knew where to start looking.
- teddyh 5y ago
  nslookup also goes directly to DNS, as far as I can tell. From what I remember, nslookup is what people used to use before “dig” became popular. So there is no good reason to ever use nslookup.
  - bradknowles 5y ago
    It depends on how your nslookup application is configured. They can be configured to go straight to DNS and not use any local host resolution, that’s true. And in that case, I consider it to be marginally useful — at least it generates less noise than dig. However, by default on most platforms, nslookup is specifically configured to use /etc/hosts (or the equivalent) or other local lookup facilities (like NIS) before using DNS. That’s the real reason you use nslookup, to see what those other local name resolution facilities might tell you that are not represented in the DNS.

Croftengea 5y ago

Also, throw in all misconfigurations related to DNSSEC and you are in hell.

bullen 5y ago

Like last time I'll post a simple snippet of Java with dns4j that solves this weeks question: >problem: Java caching DNS records forever SimpleResolver resolver = new SimpleResolver("8.8.8.8"); Name name = new Name(host + "."); Record a = Record.newRecord(name, Type.A, DClass.IN); Message query = Message.newQuery(a); Message response = resolver.send(query); Record[] answers = response.getSectionArray(Section.ANSWER);

nisa 5y ago

problem: systemd-resolved fails randomly: https://github.com/systemd/systemd/issues/19118 https://github.com/systemd/systemd/issues/19118 - this happened to me on two arch linux installations for different domains - I was unable to debug this - would love to somehow get this fixed but I switched to dnsmasq that works fine for me.

m3047 5y ago

There is the suggestion made in the article that you can get any random IDN, and to the best of my knowledge that is not true. There's something about collating characters (there is no collation order for poop, hearts or astrological signs) only, and it has to be in only one character set (no mixed languages). That said, you are free to create "<poop>.example.com", presuming you own example.com. Indeed, "`rm -rf *`.example.com" is a valid FQDN according to the DNS (protocol) as opposed to The DNS (implementation).