8 ms·

DNS resolution issue in Alpine Linux (2021)

90 points by nelsonmarcos 5y ago

rascul 5y ago
TLDR: The solution to an intentionally broken resolver is to use a third party library.
- blueflow 5y ago
  Its only broken if you dont consider robustness a feature.
  - tptacek 5y ago
    It's not just "robustness". Not supporting TCP DNS breaks DNS if your responses are "large", for values of "large" that include numbers that are in fact very small.
    - 34tlkjlaegrlk 5y ago
      largest safe size to use is ~548 bytes - anything more than that and you need tcp
      - zamadatix 5y ago
        Or, as the article says, EDNS.
    - blueflow 5y ago
      Its okay to ignore "modern" features if they create bad edge cases.
      - deleted 5y ago
        [deleted]
      - tptacek 5y ago
        This feature is from 1986.
        blueflow 5y ago
        So are absolute domain names, but everyone is using relative domain specifications now, omitting the final dot. HTTP Transfer-Encoding also got specified, and then collectively mis-implemented. That its in the standard for decades doesnt mean it will be good when used.
        charcircuit 5y ago
        I can use absolute domain names just fine in the software I use.
        tptacek 5y ago
        This doesn't make sense as an argument. Without TCP DNS, you're stuck with an untenably low limit for how much data can fit in a DNS response. Not having TCP DNS breaks DNS. It's not an aesthetic argument.
  - zamadatix 5y ago
    Things can be broken in more than one way not just solely when the thing broken is robustness (for some value of robust).
tptacek 5y ago
We're a somewhat popular hosting provider that runs Docker containers (as VMs) for our customers and does private networking over IPv6, which expands the size of our DNS requests, and we run into this all the time with Alpine. It's kind of baffling. TCP DNS is not hard. It's part of the spec. Normally, that argument doesn't mean much to me --- lots of things are parts of specs that I think are silly and not worth doing --- but TCP DNS seems like a basic necessity for DNS to work at all. What's holding this up? TCP DNS is just UDP DNS, but over a TCP connection, with the packet length sent before the packet itself. It's the simplest thing you could possibly come up with to make TCP DNS work. It's been there since the 1980s. They should add it.
- nwmcsween 5y ago
  Possibly ask on #musl or submit a patch?
  - tptacek 5y ago
    They know about it! I could write the patch, but they're not going to accept it; there's something weird going on about this.
    - nwmcsween 5y ago
      There isn't anything weird, I went on #musl asked the question, the functionality is desired but it should be worked with the community to ensure correctness.
      - rascul 5y ago
        The functionality was explicitly not added by the developer. > My choice not to do TCP in musl's stub resolver was based on an interpretation that truncated results are not just acceptable but better ux - not only do you save major round-trip delays to DNS but you also get a reasonable upper bound on # of addrs in result. https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296 Maybe he is willing to consider it now, I don't know.
traceroute66 5y ago
The moment I saw Alpine Linux in the title, my first guess was "I bet this is something to do with musl libc". Briefly looking through the blog, it looks like my gut feeling was correct. A while ago I evaluated Alpine Linux. I wanted to like it, I really did, it ticked so many boxes. But time and time again, I kept on running into issues with their adoption of musl libc. The last straw for me was when I discovered packages in their package repo (some of which were well-known names) that were compiled against musl when the upstream developers quite clearly wrote in their docs that "if you compile X against anything other than glibc, you're on your own". For me, the fact that Alpine ignored this and compiled against musl anyway, was a big red flag. (And yes I raised some of these as bug reports, but the cases got closed and nothing done about it).
- blueflow 5y ago
  "you're on your own" is like, the expected thing when you are doing something different. Still, Alpine is on top of everyone else when it comes to Docker Images sizes. Thats why it will stick.
  - traceroute66 5y ago
    > "you're on your own" is like, the expected thing when you are doing something different. Indeed. And that's fine. As long as you're willing to support that difference. But the "Alpine compiling XYZ against musl" thing is/was just being done blindly by Alpine (i.e. load X into auto-build and let it rip). Sure it compiled without errors. But it never ran properly.
    - encryptluks2 5y ago
      I've rarely had issues and if I do there is a process to report them to get them fixed. Not yet mad and act like the entire distro is worthless because I can't be bothered with reporting an issue for something that is already free. I've had less issues with Alpine than I've had with software on Windows.
  - qbasic_forever 5y ago
richardfey 5y ago
Alpine has had DNS issues since the very beginning, but I am surprised to read that it still has some.
johnklos 5y ago
There are two problems here: • musl should support EDNS and DNS over TCP/IP without issues • People should be smart enough to use DNS services that don't have stupid edge cases For the latter, if you use Google for resolving DNS, you get what you deserve. Run your own resolver if DNS resolution matters.
jve 5y ago
Few days ago, I spent quite a few hours trying to make `apk update` work for alpine on WSL2 on Windows. It didn't want to resolve dl-cdn.alpinelinux.org within alpine. Did resolve on host ubuntu. 1. WFH from VPN, firstly I had to lower mtu from 1500 to 1392 (My VPN specific issue) https://github.com/microsoft/WSL/issues/4698 https://github.com/microsoft/WSL/issues/4698 2. Next, I had to run some powershell script that updates /etc/resolv.conf to use my VPN DNS (WSL specific stuff) https://github.com/microsoft/WSL/issues/1350 https://github.com/microsoft/WSL/issues/1350 3. And I still don't know if apk works properly. Kind of works in Docker build, but I have a feeling something not quite right. See this example. Why does it "hang"? Docker command not exiting docker run -it alpine:3.15 apk update fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz Now, doing it within container itself, works: docker run -it alpine:3.15 sh / # apk update fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl- cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz v3.15.0-342-g4fee739486 [https://dl-cdn.alpinelinux.org/alpine/v3.15/main] v3.15.0-340-g4ed6115e99 [https://dl-cdn.alpinelinux.org/alpine/v3.15/community] OK: 15859 distinct packages available / # exit Can someone shed some light?
- newman314 5y ago
  Is DNS set in /etc/docker/daemon.json?
h1fra 5y ago
DNS in Alpine is notoriously buggy but it can get months until you realise that. One easy and effective solution is to force dns resolution like so dnsConfig: options: - name: ndots value: '1' cc: https://support.cloudbees.com/hc/en-us/articles/360040999471-UnknownHostException-caused-by-DNS-Resolution-issue-with-Alpine-Images https://support.cloudbees.com/hc/en-us/articles/360040999471... There are also plenty of dormant issue, enough so that I won't be using Alpine ever again imo :'(
yakubin 5y ago
> [...] the standard was extended by two options: > - Increasing the size of the UPD packet above 512 bytes via the Extension Mechanism for DNS (EDNS) > - Switching the protocol from UDP to TCP > Alpine Linux, or rather musl libc, doesn’t support either of those options. It still seems weird to me that such details are decided by libc. My reflex idea when designing a system would be to put DNS functionality in a system service, while libraries would only query the service, without troubling themselves with system caches, TCP vs UDP etc. Then possibly the service could be even swapped for another with a compatible interface, but making different decisions, without perturbing the applications. It sounds like systemd-resolved is a move in that direction, but I still don't understand why putting all that in libc, essentially making all applications perform their own independent DNS work, was the original choice.
deleted 5y ago
[deleted]
cpach 5y ago
Ouch. This kind of issue seems like a real showstopper. It really makes me hesitate using Alpine.
ComradePhil 5y ago
In my experience, the storage, bandwidth and time savings using Alpine Linux (even if they were much more significant than they are in practice) are not worth it given the issues you run into every once in a while. Just go with Ubuntu/Debian base images and you'll be much happier that you did in the long run.
brimstedt 5y ago
We've run into DNS issues with Alpine containers at two different places I've worked at. Completely different data centers and infra. First time it took a lot of effort to pinpoint the problem. Second time too, since it appeared because of a non-relevant code change (which lead to slighty more DNS requests). In both cases, a simple switch to Debian slim saved the day. Alpine is since banned from any env I'm working in :-)

90 points by nelsonmarcos 5y ago

rascul 5y ago

TLDR: The solution to an intentionally broken resolver is to use a third party library.

blueflow 5y ago
Its only broken if you dont consider robustness a feature.
- tptacek 5y ago
  It's not just "robustness". Not supporting TCP DNS breaks DNS if your responses are "large", for values of "large" that include numbers that are in fact very small.
  - 34tlkjlaegrlk 5y ago
    largest safe size to use is ~548 bytes - anything more than that and you need tcp
    - zamadatix 5y ago
      Or, as the article says, EDNS.
  - blueflow 5y ago
    Its okay to ignore "modern" features if they create bad edge cases.
    - deleted 5y ago
      [deleted]
    - tptacek 5y ago
      This feature is from 1986.
      - blueflow 5y ago
        So are absolute domain names, but everyone is using relative domain specifications now, omitting the final dot. HTTP Transfer-Encoding also got specified, and then collectively mis-implemented. That its in the standard for decades doesnt mean it will be good when used.
        charcircuit 5y ago
        I can use absolute domain names just fine in the software I use.
        tptacek 5y ago
        This doesn't make sense as an argument. Without TCP DNS, you're stuck with an untenably low limit for how much data can fit in a DNS response. Not having TCP DNS breaks DNS. It's not an aesthetic argument.
- zamadatix 5y ago
  Things can be broken in more than one way not just solely when the thing broken is robustness (for some value of robust).

tptacek 5y ago

We're a somewhat popular hosting provider that runs Docker containers (as VMs) for our customers and does private networking over IPv6, which expands the size of our DNS requests, and we run into this all the time with Alpine. It's kind of baffling. TCP DNS is not hard. It's part of the spec. Normally, that argument doesn't mean much to me --- lots of things are parts of specs that I think are silly and not worth doing --- but TCP DNS seems like a basic necessity for DNS to work at all. What's holding this up? TCP DNS is just UDP DNS, but over a TCP connection, with the packet length sent before the packet itself. It's the simplest thing you could possibly come up with to make TCP DNS work. It's been there since the 1980s. They should add it.

nwmcsween 5y ago
Possibly ask on #musl or submit a patch?
- tptacek 5y ago
  They know about it! I could write the patch, but they're not going to accept it; there's something weird going on about this.
  - nwmcsween 5y ago
    There isn't anything weird, I went on #musl asked the question, the functionality is desired but it should be worked with the community to ensure correctness.
    - rascul 5y ago
      The functionality was explicitly not added by the developer. > My choice not to do TCP in musl's stub resolver was based on an interpretation that truncated results are not just acceptable but better ux - not only do you save major round-trip delays to DNS but you also get a reasonable upper bound on # of addrs in result. https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296 Maybe he is willing to consider it now, I don't know.

traceroute66 5y ago

The moment I saw Alpine Linux in the title, my first guess was "I bet this is something to do with musl libc". Briefly looking through the blog, it looks like my gut feeling was correct. A while ago I evaluated Alpine Linux. I wanted to like it, I really did, it ticked so many boxes. But time and time again, I kept on running into issues with their adoption of musl libc. The last straw for me was when I discovered packages in their package repo (some of which were well-known names) that were compiled against musl when the upstream developers quite clearly wrote in their docs that "if you compile X against anything other than glibc, you're on your own". For me, the fact that Alpine ignored this and compiled against musl anyway, was a big red flag. (And yes I raised some of these as bug reports, but the cases got closed and nothing done about it).

blueflow 5y ago
"you're on your own" is like, the expected thing when you are doing something different. Still, Alpine is on top of everyone else when it comes to Docker Images sizes. Thats why it will stick.
- traceroute66 5y ago
  > "you're on your own" is like, the expected thing when you are doing something different. Indeed. And that's fine. As long as you're willing to support that difference. But the "Alpine compiling XYZ against musl" thing is/was just being done blindly by Alpine (i.e. load X into auto-build and let it rip). Sure it compiled without errors. But it never ran properly.
  - encryptluks2 5y ago
    I've rarely had issues and if I do there is a process to report them to get them fixed. Not yet mad and act like the entire distro is worthless because I can't be bothered with reporting an issue for something that is already free. I've had less issues with Alpine than I've had with software on Windows.
- qbasic_forever 5y ago

richardfey 5y ago

Alpine has had DNS issues since the very beginning, but I am surprised to read that it still has some.

johnklos 5y ago

There are two problems here: • musl should support EDNS and DNS over TCP/IP without issues • People should be smart enough to use DNS services that don't have stupid edge cases For the latter, if you use Google for resolving DNS, you get what you deserve. Run your own resolver if DNS resolution matters.

jve 5y ago

Few days ago, I spent quite a few hours trying to make `apk update` work for alpine on WSL2 on Windows. It didn't want to resolve dl-cdn.alpinelinux.org within alpine. Did resolve on host ubuntu. 1. WFH from VPN, firstly I had to lower mtu from 1500 to 1392 (My VPN specific issue) https://github.com/microsoft/WSL/issues/4698 https://github.com/microsoft/WSL/issues/4698 2. Next, I had to run some powershell script that updates /etc/resolv.conf to use my VPN DNS (WSL specific stuff) https://github.com/microsoft/WSL/issues/1350 https://github.com/microsoft/WSL/issues/1350 3. And I still don't know if apk works properly. Kind of works in Docker build, but I have a feeling something not quite right. See this example. Why does it "hang"? Docker command not exiting docker run -it alpine:3.15 apk update fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz Now, doing it within container itself, works: docker run -it alpine:3.15 sh / # apk update fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl- cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz v3.15.0-342-g4fee739486 [https://dl-cdn.alpinelinux.org/alpine/v3.15/main] v3.15.0-340-g4ed6115e99 [https://dl-cdn.alpinelinux.org/alpine/v3.15/community] OK: 15859 distinct packages available / # exit Can someone shed some light?

newman314 5y ago
Is DNS set in /etc/docker/daemon.json?

h1fra 5y ago

DNS in Alpine is notoriously buggy but it can get months until you realise that. One easy and effective solution is to force dns resolution like so dnsConfig: options: - name: ndots value: '1' cc: https://support.cloudbees.com/hc/en-us/articles/360040999471-UnknownHostException-caused-by-DNS-Resolution-issue-with-Alpine-Images https://support.cloudbees.com/hc/en-us/articles/360040999471... There are also plenty of dormant issue, enough so that I won't be using Alpine ever again imo :'(

yakubin 5y ago

> [...] the standard was extended by two options: > - Increasing the size of the UPD packet above 512 bytes via the Extension Mechanism for DNS (EDNS) > - Switching the protocol from UDP to TCP > Alpine Linux, or rather musl libc, doesn’t support either of those options. It still seems weird to me that such details are decided by libc. My reflex idea when designing a system would be to put DNS functionality in a system service, while libraries would only query the service, without troubling themselves with system caches, TCP vs UDP etc. Then possibly the service could be even swapped for another with a compatible interface, but making different decisions, without perturbing the applications. It sounds like systemd-resolved is a move in that direction, but I still don't understand why putting all that in libc, essentially making all applications perform their own independent DNS work, was the original choice.

deleted 5y ago

[deleted]

cpach 5y ago

Ouch. This kind of issue seems like a real showstopper. It really makes me hesitate using Alpine.

ComradePhil 5y ago

In my experience, the storage, bandwidth and time savings using Alpine Linux (even if they were much more significant than they are in practice) are not worth it given the issues you run into every once in a while. Just go with Ubuntu/Debian base images and you'll be much happier that you did in the long run.

brimstedt 5y ago

We've run into DNS issues with Alpine containers at two different places I've worked at. Completely different data centers and infra. First time it took a lot of effort to pinpoint the problem. Second time too, since it appeared because of a non-relevant code change (which lead to slighty more DNS requests). In both cases, a simple switch to Debian slim saved the day. Alpine is since banned from any env I'm working in :-)