11 ms·

Musl 1.2.4 adds TCP DNS fallback

242 points by goranmoomin 3y ago

nhatcher 3y ago
I liked seeing how Andrew Kelly/folks at ziglang supports musl. Good for them!
JdeBP 3y ago
Did the musl people never get around to implementing EDNS0? There's lots of talk here about not implementing DNS/TCP, but the musl people could have ameliorated that, and side-stepped the years of complaints to an extent, by implementing EDNS0 and allowing up to 64KiB datagrams. Yes, some (I hope more rare now than when I wrote the FGA on it almost 20 years ago) firewalls cannot handle large DNS/UDP datagrams, but this code isn't for traffic that crosses firewalls. It's for traffic that goes only as far as the local resolving proxy, or at least goes somewhere "nearby" like 9.9.9.9. EDNS0 firewall worries are a lot less here. And EDNS0, at least for the bit that enables >512 byte datagram sizes, is a lot less complex to implement than full DNS/TCP fallback. It's just proper buffer size handling and an extra resource record set.
- richardwhiuk 3y ago
  Aren't local CPEs going to be a problem for EDNS0?
richardwhiuk 3y ago
This was nearly three months ago?
- sah2ed 3y ago
  Yes. Musl 1.2.4 Released–Supports DNS Transport over TCP (openwall.com) https://news.ycombinator.com/item?id=35813978 https://news.ycombinator.com/item?id=35813978
- Arnavion 3y ago
  Yes, and for the people who link to musl dynamically in their Alpine containers, it's also in Alpine 3.18
  - yjftsjthsd-h 3y ago
    And https://alpinelinux.org/posts/Alpine-3.18.0-released.html https://alpinelinux.org/posts/Alpine-3.18.0-released.html puts that also nearly 3 months ago, FWIW.
develatio 3y ago
IIRC this was causing some exotic problems when deploying docker images based on musl.
- tyingq 3y ago
  I think there's also still some potential problems because it still does some things differently than glibc. Musl defaults to parallel requests if you define more than one nameserver (multiple --dns=, for example, for the docker daemon)...where glibc uses them in the order you provide them. To be clear, that's not "wrong", but just different maybe from what docker was expecting.
  - LaLaLand122 3y ago
    I really hope it does some things differently than glibc: https://sourceware.org/bugzilla/show_bug.cgi?id=19643 https://sourceware.org/bugzilla/show_bug.cgi?id=19643 In any case glibc uses NSS, so what glibc does depends on the configuration. It may well just forward the request to systemd-resolved.
InvaderFizz 3y ago
Glad to see this finally come to fruition. This has been an issue plaguing Alpine for years where the musl maintainer basically said the standard says may fallback, not must fallback. Let the rest of the internet change, we don't feel this is important. We're standards complainant. It gained traction for the change with the latest RFC. for dns last year which made TCP fallback mandatory [0]. The cloak of standards compliant could no longer be used. 0: https://datatracker.ietf.org/doc/html/rfc9210 https://datatracker.ietf.org/doc/html/rfc9210
- vidarh 3y ago
  Yikes. The first time I ran into an issue where email delivery for a major provider was impossible without TCP fallback was 23 years ago. To treat this as optional this long is ridiculous.
  - brmgb 3y ago
    For all the good things Musl does, its authors can often be downright ridiculous out of a dogmatism which is very unpleasant to deal with as a user. See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling.
    - 0xcde4c3db 3y ago
      > See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling. I suspect that the primary use cases for that would be applying half-baked workarounds and refusing to work with an "unsupported" libc, much like HTTP User-Agent.
      - brmgb 3y ago
        Or, you know, taking into account the stupid things Musl does like not having TCP DNS fallback. I’m only half joking. Features detection in Musl is a generally pain. I don’t blame developers for sometimes taking the path of least resistance and not trying to support it.
robinhoodexe 3y ago
We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70% of our 500 container images being used for Python, R or node the final image is so large (due to libraries/packages), that the difference between alpine (~30 MB) and debian-slim (~80) is negligible. We’ve been experiencing the weird DNS behaviour of alpine and other issues with musl as well. Debian is rock solid and upgrading to bookworm from bullseye and even buster in many cases didn’t cause any problems at all. I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp). It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.
- amouat 3y ago
  You might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc. Our images come without a shell or pacakge manager by default, but there are -dev variants that include these. https://github.com/wolfi-dev/ https://github.com/wolfi-dev/ https://github.com/chainguard-images/images https://github.com/chainguard-images/images
  - kapilvt 3y ago
    I was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it. https://www.chainguard.dev/unchained/scaling-chainguard-images-with-a-growing-catalog-and-proactive-security-updates https://www.chainguard.dev/unchained/scaling-chainguard-imag... Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.
deleted 3y ago
[deleted]
joelverhagen 3y ago
Really happy to see this. This caused random NuGet package restore issues when the CNAME chain for api.nuget.org exceeded a certain length. https://github.com/NuGet/NuGetGallery/issues/9396 https://github.com/NuGet/NuGetGallery/issues/9396 Our CDN provider ended up having a shedding mode in some hot areas that made the chain exceed the limit from time to time. Our multi CDN set up saved us so we could do geo specific failovers.
ecliptik 3y ago
While I'm glad this is finally addressed, this limits the usefulness of one of my favorite interview questions. Asking about Alpine in a production environment was always good way finding who has container experiences of watching C-Beams glitter in the dark to those who only just read a "10 Docker Container Tricks; #8 will blow your mind!" blog post from 2017.
- InvaderFizz 3y ago
  It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it. DNS is a fun rabbit hole for interviews, for sure. My favorite one to see on a resume is NIS. If you are listing NIS and don't have horror stories or other things to say about NIS, that's a really good indicator of the value of your resume. I intentionally list NIS on my resume because it is such a fun conversation topic to go on about how security models changed over time, all the ways NIS is terrible, but also how simple and useful it was.
  - meroje 3y ago
    > It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it. I'm still not going back until I can configure for TCP to be the default.
  - geocar 3y ago
    I still use NIS because hosts files are faster than DNS.
  - ecliptik 3y ago
    NIS is a good one. I have UUCP as a skill on my resume as an easter egg but no one ever asks about it. For DNS, my favorite interview question goes like this, How would you verify DNS is resolving from within a pod on Kubernetes? After listening to the answer, add some constraints: 1. Common networking utilities like ping, nslookup, dig, etc are not available 2. Container user is unpriviledged 3. su/sudo do not work This can lead to some elaborate k8s troubleshooting or the simple, and correct, answer of getent hosts.
jake_morrison 3y ago
I use distroless images based on Debian or Ubuntu, e.g., https://github.com/cogini/phoenix_container_example https://github.com/cogini/phoenix_container_example The result is images the same size as Alpine, or smaller, without the incompatibilities. I think Alpine is a dead end.
- nullify88 3y ago
  Doesn't distroless bring in a lot of complexity when you need something as simple as ca-certificates? IMO Distroless or even scratch is nice for statically complied binaries or self contained deployments, but if there's a dependency on user space then it becomes complex.
  - pas 3y ago
    You can copy the CA bundle in the last step of the image build (along with the required libs and their assets/dependencies), or am I missing something?
- suprjami 3y ago
  I hadn't heard of "distroless" before. Confusing name for a container with just main process runtimes, but neat idea. https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
deleted 3y ago
[deleted]
nathants 3y ago
i’ve stopped using containers because they are annoying. i’ve started using alpine on my laptop and ec2 because it’s not. different strokes, different folks.
saagarjha 3y ago
This seems pretty neat. We’ve been getting reports from a lot of people where they seem to be having random DNS failures, and from what we can tell it’s because they’re on an IPv6-only network (seems very common in developing countries with infrastructure that can’t hand out IPv4) and the UDP DNS fails. Hope this helps resolve it for them.

242 points by goranmoomin 3y ago

I liked seeing how Andrew Kelly/folks at ziglang supports musl. Good for them!

Did the musl people never get around to implementing EDNS0? There's lots of talk here about not implementing DNS/TCP, but the musl people could have ameliorated that, and side-stepped the years of complaints to an extent, by implementing EDNS0 and allowing up to 64KiB datagrams. Yes, some (I hope more rare now than when I wrote the FGA on it almost 20 years ago) firewalls cannot handle large DNS/UDP datagrams, but this code isn't for traffic that crosses firewalls. It's for traffic that goes only as far as the local resolving proxy, or at least goes somewhere "nearby" like 9.9.9.9. EDNS0 firewall worries are a lot less here. And EDNS0, at least for the bit that enables >512 byte datagram sizes, is a lot less complex to implement than full DNS/TCP fallback. It's just proper buffer size handling and an extra resource record set.

richardwhiuk 3y ago
Aren't local CPEs going to be a problem for EDNS0?

richardwhiuk 3y ago

This was nearly three months ago?

sah2ed 3y ago
Yes. Musl 1.2.4 Released–Supports DNS Transport over TCP (openwall.com) https://news.ycombinator.com/item?id=35813978 https://news.ycombinator.com/item?id=35813978
Arnavion 3y ago
Yes, and for the people who link to musl dynamically in their Alpine containers, it's also in Alpine 3.18
- yjftsjthsd-h 3y ago
  And https://alpinelinux.org/posts/Alpine-3.18.0-released.html https://alpinelinux.org/posts/Alpine-3.18.0-released.html puts that also nearly 3 months ago, FWIW.

develatio 3y ago

IIRC this was causing some exotic problems when deploying docker images based on musl.

tyingq 3y ago
I think there's also still some potential problems because it still does some things differently than glibc. Musl defaults to parallel requests if you define more than one nameserver (multiple --dns=, for example, for the docker daemon)...where glibc uses them in the order you provide them. To be clear, that's not "wrong", but just different maybe from what docker was expecting.
- LaLaLand122 3y ago
  I really hope it does some things differently than glibc: https://sourceware.org/bugzilla/show_bug.cgi?id=19643 https://sourceware.org/bugzilla/show_bug.cgi?id=19643 In any case glibc uses NSS, so what glibc does depends on the configuration. It may well just forward the request to systemd-resolved.

InvaderFizz 3y ago

Glad to see this finally come to fruition. This has been an issue plaguing Alpine for years where the musl maintainer basically said the standard says may fallback, not must fallback. Let the rest of the internet change, we don't feel this is important. We're standards complainant. It gained traction for the change with the latest RFC. for dns last year which made TCP fallback mandatory [0]. The cloak of standards compliant could no longer be used. 0: https://datatracker.ietf.org/doc/html/rfc9210 https://datatracker.ietf.org/doc/html/rfc9210

vidarh 3y ago
Yikes. The first time I ran into an issue where email delivery for a major provider was impossible without TCP fallback was 23 years ago. To treat this as optional this long is ridiculous.
- brmgb 3y ago
  For all the good things Musl does, its authors can often be downright ridiculous out of a dogmatism which is very unpleasant to deal with as a user. See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling.
  - 0xcde4c3db 3y ago
    > See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling. I suspect that the primary use cases for that would be applying half-baked workarounds and refusing to work with an "unsupported" libc, much like HTTP User-Agent.
    - brmgb 3y ago
      Or, you know, taking into account the stupid things Musl does like not having TCP DNS fallback. I’m only half joking. Features detection in Musl is a generally pain. I don’t blame developers for sometimes taking the path of least resistance and not trying to support it.

robinhoodexe 3y ago

We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70% of our 500 container images being used for Python, R or node the final image is so large (due to libraries/packages), that the difference between alpine (~30 MB) and debian-slim (~80) is negligible. We’ve been experiencing the weird DNS behaviour of alpine and other issues with musl as well. Debian is rock solid and upgrading to bookworm from bullseye and even buster in many cases didn’t cause any problems at all. I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp). It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.

amouat 3y ago
You might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc. Our images come without a shell or pacakge manager by default, but there are -dev variants that include these. https://github.com/wolfi-dev/ https://github.com/wolfi-dev/ https://github.com/chainguard-images/images https://github.com/chainguard-images/images
- kapilvt 3y ago
  I was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it. https://www.chainguard.dev/unchained/scaling-chainguard-images-with-a-growing-catalog-and-proactive-security-updates https://www.chainguard.dev/unchained/scaling-chainguard-imag... Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.

deleted 3y ago

[deleted]

joelverhagen 3y ago

Really happy to see this. This caused random NuGet package restore issues when the CNAME chain for api.nuget.org exceeded a certain length. https://github.com/NuGet/NuGetGallery/issues/9396 https://github.com/NuGet/NuGetGallery/issues/9396 Our CDN provider ended up having a shedding mode in some hot areas that made the chain exceed the limit from time to time. Our multi CDN set up saved us so we could do geo specific failovers.

ecliptik 3y ago

While I'm glad this is finally addressed, this limits the usefulness of one of my favorite interview questions. Asking about Alpine in a production environment was always good way finding who has container experiences of watching C-Beams glitter in the dark to those who only just read a "10 Docker Container Tricks; #8 will blow your mind!" blog post from 2017.

InvaderFizz 3y ago
It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it. DNS is a fun rabbit hole for interviews, for sure. My favorite one to see on a resume is NIS. If you are listing NIS and don't have horror stories or other things to say about NIS, that's a really good indicator of the value of your resume. I intentionally list NIS on my resume because it is such a fun conversation topic to go on about how security models changed over time, all the ways NIS is terrible, but also how simple and useful it was.
- meroje 3y ago
  > It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it. I'm still not going back until I can configure for TCP to be the default.
- geocar 3y ago
  I still use NIS because hosts files are faster than DNS.
- ecliptik 3y ago
  NIS is a good one. I have UUCP as a skill on my resume as an easter egg but no one ever asks about it. For DNS, my favorite interview question goes like this, How would you verify DNS is resolving from within a pod on Kubernetes? After listening to the answer, add some constraints: 1. Common networking utilities like ping, nslookup, dig, etc are not available 2. Container user is unpriviledged 3. su/sudo do not work This can lead to some elaborate k8s troubleshooting or the simple, and correct, answer of getent hosts.

jake_morrison 3y ago

I use distroless images based on Debian or Ubuntu, e.g., https://github.com/cogini/phoenix_container_example https://github.com/cogini/phoenix_container_example The result is images the same size as Alpine, or smaller, without the incompatibilities. I think Alpine is a dead end.

nullify88 3y ago
Doesn't distroless bring in a lot of complexity when you need something as simple as ca-certificates? IMO Distroless or even scratch is nice for statically complied binaries or self contained deployments, but if there's a dependency on user space then it becomes complex.
- pas 3y ago
  You can copy the CA bundle in the last step of the image build (along with the required libs and their assets/dependencies), or am I missing something?
suprjami 3y ago
I hadn't heard of "distroless" before. Confusing name for a container with just main process runtimes, but neat idea. https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless

deleted 3y ago

[deleted]

nathants 3y ago

i’ve stopped using containers because they are annoying. i’ve started using alpine on my laptop and ec2 because it’s not. different strokes, different folks.

saagarjha 3y ago

This seems pretty neat. We’ve been getting reports from a lot of people where they seem to be having random DNS failures, and from what we can tell it’s because they’re on an IPv6-only network (seems very common in developing countries with infrastructure that can’t hand out IPv4) and the UDP DNS fails. Hope this helps resolve it for them.