10 ms·

Go 1.21 will (likely) have a static toolchain on Linux

190 points by ingve 4y ago

ZiiS 4y ago
Whilst this looses some exhotic /etc/resolv.conf compatibility it is a step towards a fully reproducable toolchain which seems a much bigger win.
- dgl 4y ago
  It sounds like it's pretty much the final step; the toolchain will be fully reproducible on 1.21: https://go.dev/cl/454836 https://go.dev/cl/454836 The CL description describes the changes, then: "Combined, these four changes (along with Go 1.20's removal of installed pkg/*.a files and conversion of macOS net away from cgo) make the output of make.bash fully reproducible, even when cross-compiling: a released macOS toolchain built on Linux or Windows will contain exactly the same bits as a released macOS toolchain built on macOS. The word "released" in the previous sentence is important. For the build IDs in the binaries to work out the same on both systems, a VERSION file must exist to provide a consistent compiler build ID (instead of using a content hash of the binary)."
benatkin 4y ago
That's nice for Alpine isn't it? That uses MUSL. Speaking of portability and golang, this sounds promising: https://github.com/tetratelabs/wazero https://github.com/tetratelabs/wazero Can anyone tell me if this is close to working as well as wasmtime, which uses CGO?
- jdub 4y ago
  What the article describes is a compiled Go binary that will run on any* Linux system without using any form of libc, dynamically or statically. So it's not just relevant to musl or musl-based distributions (perhaps especially because musl is almost always linked statically anyway, so the distribution doesn't really matter in that case either). * albeit with a minimum kernel version
  - elnatro 4y ago
    Wow, so the best language for multi platform Linux development could be Golang. Does it has enough libraries to be a serious alternative? What do you think about that?
    - ptspts 4y ago
      The zig tool can also do this conveniently for programming languages C and Zig.
  - ungamedplayer 4y ago
    So these apps use their own resolver, where do they get settings from?
    - benatkin 4y ago
      Getting rid of the settings is seen as a plus, I think. It's predictable behavior. Except it may detect some.
      - jeremyjh 4y ago
        I would have predicted that an app using the network on my computer would make use of my computer’s network settings, since all other apps do. How would it know the correct resolver to use on my network otherwise?
        benatkin 4y ago
        The answer is in another thread, go only checks resolv.conf. Other settings get ignored. That means if you see unexpected behavior on one system you only have to check resolv.conf, not all the other network settings.
    - mkup 4y ago
      I guess they parse /etc/resolv.conf (via Go standard library) and just ignore /etc/nsswitch.conf (which is going to work OK in 99.999% cases). For the rest 0.001% cases there is a non-default option to link to system libc dynamically.
baby 4y ago
As a noob. Why do programming languages rely on libc, why not a Golang library instead? And why do programs don’t statically link to libc in general?
- silisili 4y ago
  In Go's case, because DNS is such a bear. Not the looking up part, but all the sources and behaviors behind where and how to lookup. IIRC they couldn't even do it natively on MacOS due to system restrictions, unsure if that's still the case.
  - cpach 4y ago
    Native DNS lookups is possible in Go on macOS. But it seems the road to get there was not exactly smooth.
    - mseepgood 4y ago
      It's not difficult, and it was done like that for some time, but the macOS system call interface is not guaranteed to be stable (more like guaranteed to be unstable), so the correct thing on macOS is to go through libc (libSystem), which Go now does since 1.12.
      - cpach 4y ago
        Not exactly rocket science perhaps, but AFAICT there was some long-running tickets related to DNS lookups on macOS.
      - jen20 4y ago
        For DNS specifically, this landed in Go 1.20: here is the change set [1]. A different change [2] had a similar effect of using the system facilities for certificate validation in 1.18. In general, using the custom stuff “works” (until it doesn’t!) but results in janky programs that don’t behave with respect to split tunnel DNS and so forth - very common with cross compiled programs until recently. [1]: https://go-review.googlesource.com/c/go/+/446178 https://go-review.googlesource.com/c/go/+/446178 [2]: https://go-review.googlesource.com/c/go/+/353132 https://go-review.googlesource.com/c/go/+/353132
bsaul 4y ago
Coming from swift, it's pretty refreshing to see a language that has decided to settle on the language itself pretty early and focused on improving all other aspects of the dev experience. Last time i ran my go test suite on the CI i thought it had a bug because it was so fast compared to what i'm used to with swift. I do still have a weird feeling of getting back to the stone age whenever i get a nil pointer panic though. i wish go devs could figure out a way of fixing those last quirks without impacting the rest too much.
- kramerger 4y ago
  In this regard Go has been the exact opposite of Rust.
  - diarrhea 4y ago
    Isn’t the Rust DX pretty good still?
    - akmittal 4y ago
      Rust has very long compile time compared to Go
      - dgunay 4y ago
        Rust could also have shorter compile times, if it gave up on a lot of the optimizations and features it currently does.
        eptcyka 4y ago
        A massive source of compilation pain is that Rust just generally generates a lot of code - it always prefers specialization over dynamic dispatch unless the source specificies dynamic dispatch. Couple this with a culture of believing dynamic dispatch is always worse and lots of codegen for generics and you get one hell of a task for your linter. I wouldn't call this an optimization, but rather an almost cultural design decision.
imoverclocked 4y ago
I really hope someone slips an ascii art Delorian somewhere into the release.
weitzj 4y ago
Edit: More context for the following text: https://utcc.utoronto.ca/~cks/space/blog/programming/GoWhyNotStaticLinked https://utcc.utoronto.ca/~cks/space/blog/programming/GoWhyNo... Interesting from a project I currently work on, we use a static compiled binary and I thought just the DNS resolver was a thing which needs to be switched and use the internal DNS resolver of the Go sdk. But as we noted, once you have something like PAM plugins and using ldap sssd to get users from your LDAP, the static Go binary cannot resolve the user IDs. So there is more to it and we have to decide whether we implement a user lookup natively in the Go binary or rely on libc and how the actual Linux system is configured to login users. Also I have not tested this hypothesis, but I assume that with all the troubles I have seen with a Linux vpn client pushing their resolvers into the Linux client and how systemd resolver and other kinds of resolvers mess up your DNS, I wonder if the Go internal DNS resolver implementation picks up on this mess. Anyways for the Go Compiler here this might not be such a big problem.
- XorNot 4y ago
  DNS is pretty straight forward though, because systemd-resolved runs the stub-resolver. So if your /etc/resolv.conf points to it (as it usually will) then everything will "just work" fine since it's all DNS requests. What we really need is something similar for doing users and groups when you get down to it: which we should have because at the end of the day we're really just asking the system tell us some UIDs and GIDs, or what names to assign to such things.
  - arccy 4y ago
    The answer is, of course, more systemd: https://go.dev/issue/38810 https://go.dev/issue/38810
  - aragilar 4y ago
    We do, it's nss (which you can configure via nsswitch.conf), which is what the go toolchain is no longer compiling against.
    - XorNot 4y ago
zoobab 4y ago
I have a project to rebuild a complete alpine to make static binaries, glibc does not like static for political reasons, the feature to make static binaries is broken, and undocumented on purpose. Long life musl! http://stalinux.wikidot.com http://stalinux.wikidot.com
- mort96 4y ago
  GNU may have a political opposition to static linking, but there are major practical issues too. For features like nsswitch, the libc has to dlopen shared libraries, which themselves link against libc. Static linking means you'll end up with (at least) 2 different libc versions in the address space at the same time, operating on the same data structures. GNU (and some other libc implementors) try to make that work as well as possible, but it's necessarily an unholy mess.
  - zoobab 4y ago
    28 years of using linux, never used nsswitch.conf. I learned about it's existence when a colleague of mine asked a question about it during an recruiting interview of a candidate.
    - jackweirdy 4y ago
      I feel like someone has opened a jet engine and showed me there’s a hamster wheel in there; I had no idea this even existed
    - InvaderFizz 4y ago
      Just a matter of where and how you worked. I would expect anyone doing sysadmin work on Linux in any sort of centrally managed network would have exposure to it. It is certainly relevant any time you are messing with NIS/LDAP configs and troubleshooting. Your typical developer, even doing C app development I wouldn't expect to have interacted with nsswitch unless their local name resolution was broken somehow and they were trying to fix their own problems.
      - 4y ago
awesomegoat_com 4y ago
Are we surely going in the right direction?
- awithrow 4y ago
  what are your concerns here?
speakspokespok 4y ago
> (You might wonder why I didn't just … I needed to run on … with … in the first place. The answer is that I was distracted by the flow of circumstances. First I tried … so I tried to …, then I had … same problem, and by that point my mental focus was on 'make the compiler toolchain work'.) While intimately aware of the phenomenon in a technical context and something I constantly check myself on, I’ve never heard it described quite like that. ‘distracted by the flow of circumstances.’ So relevant to many parts of one’s life, it’s a nice turn of phrase, yet also complete BS. I learned a lot from this article and discussion here!
kaulquappe 4y ago
We build our client executables in a debian squeeze docker container exactly because of those glibc versioning issues. This way, they run on all currently supported linux distributions, even on RHEL 6.
weitzj 4y ago
Also interesting project to point out since we discuss DNS and LDAP, nsswitch here: A library for calling C functions from Go without Cgo. https://github.com/ebitengine/purego https://github.com/ebitengine/purego https://news.ycombinator.com/item?id=34763681 https://news.ycombinator.com/item?id=34763681
hamdouni 4y ago
That will surely help in my heterogenous servers environment with multiple OSes ... No more need to use Docker as a builder !

190 points by ingve 4y ago

ZiiS 4y ago

Whilst this looses some exhotic /etc/resolv.conf compatibility it is a step towards a fully reproducable toolchain which seems a much bigger win.

dgl 4y ago
It sounds like it's pretty much the final step; the toolchain will be fully reproducible on 1.21: https://go.dev/cl/454836 https://go.dev/cl/454836 The CL description describes the changes, then: "Combined, these four changes (along with Go 1.20's removal of installed pkg/*.a files and conversion of macOS net away from cgo) make the output of make.bash fully reproducible, even when cross-compiling: a released macOS toolchain built on Linux or Windows will contain exactly the same bits as a released macOS toolchain built on macOS. The word "released" in the previous sentence is important. For the build IDs in the binaries to work out the same on both systems, a VERSION file must exist to provide a consistent compiler build ID (instead of using a content hash of the binary)."

benatkin 4y ago

That's nice for Alpine isn't it? That uses MUSL. Speaking of portability and golang, this sounds promising: https://github.com/tetratelabs/wazero https://github.com/tetratelabs/wazero Can anyone tell me if this is close to working as well as wasmtime, which uses CGO?

jdub 4y ago
What the article describes is a compiled Go binary that will run on any* Linux system without using any form of libc, dynamically or statically. So it's not just relevant to musl or musl-based distributions (perhaps especially because musl is almost always linked statically anyway, so the distribution doesn't really matter in that case either). * albeit with a minimum kernel version
- elnatro 4y ago
  Wow, so the best language for multi platform Linux development could be Golang. Does it has enough libraries to be a serious alternative? What do you think about that?
  - ptspts 4y ago
    The zig tool can also do this conveniently for programming languages C and Zig.
- ungamedplayer 4y ago
  So these apps use their own resolver, where do they get settings from?
  - benatkin 4y ago
    Getting rid of the settings is seen as a plus, I think. It's predictable behavior. Except it may detect some.
    - jeremyjh 4y ago
      I would have predicted that an app using the network on my computer would make use of my computer’s network settings, since all other apps do. How would it know the correct resolver to use on my network otherwise?
      - benatkin 4y ago
        The answer is in another thread, go only checks resolv.conf. Other settings get ignored. That means if you see unexpected behavior on one system you only have to check resolv.conf, not all the other network settings.
  - mkup 4y ago
    I guess they parse /etc/resolv.conf (via Go standard library) and just ignore /etc/nsswitch.conf (which is going to work OK in 99.999% cases). For the rest 0.001% cases there is a non-default option to link to system libc dynamically.

baby 4y ago

As a noob. Why do programming languages rely on libc, why not a Golang library instead? And why do programs don’t statically link to libc in general?

silisili 4y ago
In Go's case, because DNS is such a bear. Not the looking up part, but all the sources and behaviors behind where and how to lookup. IIRC they couldn't even do it natively on MacOS due to system restrictions, unsure if that's still the case.
- cpach 4y ago
  Native DNS lookups is possible in Go on macOS. But it seems the road to get there was not exactly smooth.
  - mseepgood 4y ago
    It's not difficult, and it was done like that for some time, but the macOS system call interface is not guaranteed to be stable (more like guaranteed to be unstable), so the correct thing on macOS is to go through libc (libSystem), which Go now does since 1.12.
    - cpach 4y ago
      Not exactly rocket science perhaps, but AFAICT there was some long-running tickets related to DNS lookups on macOS.
    - jen20 4y ago
      For DNS specifically, this landed in Go 1.20: here is the change set [1]. A different change [2] had a similar effect of using the system facilities for certificate validation in 1.18. In general, using the custom stuff “works” (until it doesn’t!) but results in janky programs that don’t behave with respect to split tunnel DNS and so forth - very common with cross compiled programs until recently. [1]: https://go-review.googlesource.com/c/go/+/446178 https://go-review.googlesource.com/c/go/+/446178 [2]: https://go-review.googlesource.com/c/go/+/353132 https://go-review.googlesource.com/c/go/+/353132

bsaul 4y ago

Coming from swift, it's pretty refreshing to see a language that has decided to settle on the language itself pretty early and focused on improving all other aspects of the dev experience. Last time i ran my go test suite on the CI i thought it had a bug because it was so fast compared to what i'm used to with swift. I do still have a weird feeling of getting back to the stone age whenever i get a nil pointer panic though. i wish go devs could figure out a way of fixing those last quirks without impacting the rest too much.

kramerger 4y ago
In this regard Go has been the exact opposite of Rust.
- diarrhea 4y ago
  Isn’t the Rust DX pretty good still?
  - akmittal 4y ago
    Rust has very long compile time compared to Go
    - dgunay 4y ago
      Rust could also have shorter compile times, if it gave up on a lot of the optimizations and features it currently does.
      - eptcyka 4y ago
        A massive source of compilation pain is that Rust just generally generates a lot of code - it always prefers specialization over dynamic dispatch unless the source specificies dynamic dispatch. Couple this with a culture of believing dynamic dispatch is always worse and lots of codegen for generics and you get one hell of a task for your linter. I wouldn't call this an optimization, but rather an almost cultural design decision.

imoverclocked 4y ago

I really hope someone slips an ascii art Delorian somewhere into the release.

weitzj 4y ago

Edit: More context for the following text: https://utcc.utoronto.ca/~cks/space/blog/programming/GoWhyNotStaticLinked https://utcc.utoronto.ca/~cks/space/blog/programming/GoWhyNo... Interesting from a project I currently work on, we use a static compiled binary and I thought just the DNS resolver was a thing which needs to be switched and use the internal DNS resolver of the Go sdk. But as we noted, once you have something like PAM plugins and using ldap sssd to get users from your LDAP, the static Go binary cannot resolve the user IDs. So there is more to it and we have to decide whether we implement a user lookup natively in the Go binary or rely on libc and how the actual Linux system is configured to login users. Also I have not tested this hypothesis, but I assume that with all the troubles I have seen with a Linux vpn client pushing their resolvers into the Linux client and how systemd resolver and other kinds of resolvers mess up your DNS, I wonder if the Go internal DNS resolver implementation picks up on this mess. Anyways for the Go Compiler here this might not be such a big problem.

XorNot 4y ago
DNS is pretty straight forward though, because systemd-resolved runs the stub-resolver. So if your /etc/resolv.conf points to it (as it usually will) then everything will "just work" fine since it's all DNS requests. What we really need is something similar for doing users and groups when you get down to it: which we should have because at the end of the day we're really just asking the system tell us some UIDs and GIDs, or what names to assign to such things.
- arccy 4y ago
  The answer is, of course, more systemd: https://go.dev/issue/38810 https://go.dev/issue/38810
- aragilar 4y ago
  We do, it's nss (which you can configure via nsswitch.conf), which is what the go toolchain is no longer compiling against.
  - XorNot 4y ago

zoobab 4y ago

I have a project to rebuild a complete alpine to make static binaries, glibc does not like static for political reasons, the feature to make static binaries is broken, and undocumented on purpose. Long life musl! http://stalinux.wikidot.com http://stalinux.wikidot.com

mort96 4y ago
GNU may have a political opposition to static linking, but there are major practical issues too. For features like nsswitch, the libc has to dlopen shared libraries, which themselves link against libc. Static linking means you'll end up with (at least) 2 different libc versions in the address space at the same time, operating on the same data structures. GNU (and some other libc implementors) try to make that work as well as possible, but it's necessarily an unholy mess.
- zoobab 4y ago
  28 years of using linux, never used nsswitch.conf. I learned about it's existence when a colleague of mine asked a question about it during an recruiting interview of a candidate.
  - jackweirdy 4y ago
    I feel like someone has opened a jet engine and showed me there’s a hamster wheel in there; I had no idea this even existed
  - InvaderFizz 4y ago
    Just a matter of where and how you worked. I would expect anyone doing sysadmin work on Linux in any sort of centrally managed network would have exposure to it. It is certainly relevant any time you are messing with NIS/LDAP configs and troubleshooting. Your typical developer, even doing C app development I wouldn't expect to have interacted with nsswitch unless their local name resolution was broken somehow and they were trying to fix their own problems.
    - 4y ago

awesomegoat_com 4y ago

Are we surely going in the right direction?

awithrow 4y ago
what are your concerns here?

speakspokespok 4y ago

> (You might wonder why I didn't just … I needed to run on … with … in the first place. The answer is that I was distracted by the flow of circumstances. First I tried … so I tried to …, then I had … same problem, and by that point my mental focus was on 'make the compiler toolchain work'.) While intimately aware of the phenomenon in a technical context and something I constantly check myself on, I’ve never heard it described quite like that. ‘distracted by the flow of circumstances.’ So relevant to many parts of one’s life, it’s a nice turn of phrase, yet also complete BS. I learned a lot from this article and discussion here!

kaulquappe 4y ago

We build our client executables in a debian squeeze docker container exactly because of those glibc versioning issues. This way, they run on all currently supported linux distributions, even on RHEL 6.

weitzj 4y ago

Also interesting project to point out since we discuss DNS and LDAP, nsswitch here: A library for calling C functions from Go without Cgo. https://github.com/ebitengine/purego https://github.com/ebitengine/purego https://news.ycombinator.com/item?id=34763681 https://news.ycombinator.com/item?id=34763681

hamdouni 4y ago

That will surely help in my heterogenous servers environment with multiple OSes ... No more need to use Docker as a builder !