3 ms·

> the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back Can you expand on this? It’s an interesting perspect

> the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back Can you expand on this? It’s an interesting perspective I haven’t heard before. I get the gist but even without a Dockerfile, a container image is much less opaque and much more deterministic than a binary with a statically linked library where the compiler might inline entire functions and perform optimizations across module boundaries with LTO. It’s not as simple as setting LD_LIBRARY_PATH but it’s not like trying to reverse the information lost during compilation.

matheusmoreira 4y ago
> Can you expand on this? Bundling dependecies is essentially static linking that doesn't violate the GPL. Static linking means all dependencies are compiled into the binary. Modern technology like docker, appimages, snaps, etc. all emulate that by creating a static image with all the dependencies and then dynamically linking to those specific libraries. So now we have a secondary linker and loader on top of the operating system's.
- benreesman 4y ago
  And it' such a shame because GNU/FSF was like the friggin Rebel Alliance fighting the Empire for the longest time: we'd never be where we are today without that stuff. Those folks were my heroes growing up. But power corrupts, and once you've got everyone linking to your `libc`, the temptation to get some combination of lazy around standards compliance at best, and seduced by what amounts to proprietary vendor API lock-in to push your politics in at least some cases seemingly becomes overwhelming. Add in LLVM kicking your ass from here to FreeBSD with Stallman freaking the fuck out about `gcc`'s plunging relevance and there's understandable but still not acceptable instinct to go further into incompatible-by-design. People on here routinely lose their shit because NVIDIA or whoever makes it a nightmare to interoperate, and I don't like it either, but vendor lock-in is vendor lock-in. `emacs` talks to the LSP servers via JSON for Christ's sake, which only makes it about 100x laggier, but you can see why they did it, because just like Docker even RMS can't convince anyone that What We Went With is "linking". God bless the VSCode people for accepting the performance penalty so that the rest of us can come to the party, they could have muscled LSP out of the way IMO and didn't. So `rust-analyzer` gets to shovel megabytes of JSON around and Docker gets to (figuratively) tarball a bunch of `.so` files so that `ldd` can "dynamically" link them. The thing is, this way lies irrelevance. Even the GNU userland is pretty optional these days. Is Alpine, as I've taken to calling it, "GNU plus Linux"? Is NixOS? What are we putting up with this shit for? Bash?
  - matheusmoreira 4y ago
    The history of free software is the history of the GPL and the power struggles around it. Free software developers neglecting binary interfaces in favor of source code, refusing to support plugins because it enables proprietary applications, new compilers arising to challenge GCC as a result, hardware makers using cryptography to prevent users from running their own code... And now we have this "hybrid linking" thing: static image containing dynamically linked libraries.
benreesman 4y ago
There's a big subthread on this post about it: https://news.ycombinator.com/item?id=32459755 https://news.ycombinator.com/item?id=32459755. Hermetic builds (Bazel, Nix, etc.) are in some sense orthogonal to packaging and delivery via e.g. a Docker container. It's true that it takes great care to get bit-for-bit identical output from a C/C++ toolchain, and it's also true that very subtle things can send the compiler and linker into a tailspin of "now I inline this and now I don't have room to inline that and now other optimization passes do something different" and whoa, my program just got a lot faster or slower, yikes. I forget the title now, but there was a great Strange Loop talk about how if you do good statistics on repeated builds or even just stuff more or less into `argv` that the difference between `-O2` and `-O3` doesn't hold up (on either `gcc` or `clang`). But whether the final executable is delivered to the target system via Docker or `scp` doesn't change any of that. If you want identical builds the Bazel and Nix people have a high-investment/high-return sales pitch they'd like you to hear :). And it's also true (as another commenter pointed out) that it's at least a little bit reductionist of me to say "Docker gives you static linking". Docker does a zillion things, and for the people using the other stuff, more power to them if it's working well. I would contend that most of them (especially since QEMU emulating x86_64 on your Intel Mac or vice versa isn't really a great story anymore) are not the real selling point though: you already had `virtualenv` and `nvm` and `rustup` and just countless ways to isolate/reproduce everything except the contents of your friendly neighborhood `/usr/lib/...`. If the native compiled code underpinning all of this stuff had all it's symbols linked in, and your kernel is the same or newer, you could use a tarball (and I've worked places where we did just that)! `git` was probably the tool that really sold everyone on: I want a goddamned cryptographic hash of everything in the thing, otherwise I Don't Trust Like That. We don't accept mystery meat from our source code anymore, why we feel any differently about what actually runs is Stockholm's syndrome IMHO.

matheusmoreira 4y ago

> Can you expand on this? Bundling dependecies is essentially static linking that doesn't violate the GPL. Static linking means all dependencies are compiled into the binary. Modern technology like docker, appimages, snaps, etc. all emulate that by creating a static image with all the dependencies and then dynamically linking to those specific libraries. So now we have a secondary linker and loader on top of the operating system's.

benreesman 4y ago
And it' such a shame because GNU/FSF was like the friggin Rebel Alliance fighting the Empire for the longest time: we'd never be where we are today without that stuff. Those folks were my heroes growing up. But power corrupts, and once you've got everyone linking to your `libc`, the temptation to get some combination of lazy around standards compliance at best, and seduced by what amounts to proprietary vendor API lock-in to push your politics in at least some cases seemingly becomes overwhelming. Add in LLVM kicking your ass from here to FreeBSD with Stallman freaking the fuck out about `gcc`'s plunging relevance and there's understandable but still not acceptable instinct to go further into incompatible-by-design. People on here routinely lose their shit because NVIDIA or whoever makes it a nightmare to interoperate, and I don't like it either, but vendor lock-in is vendor lock-in. `emacs` talks to the LSP servers via JSON for Christ's sake, which only makes it about 100x laggier, but you can see why they did it, because just like Docker even RMS can't convince anyone that What We Went With is "linking". God bless the VSCode people for accepting the performance penalty so that the rest of us can come to the party, they could have muscled LSP out of the way IMO and didn't. So `rust-analyzer` gets to shovel megabytes of JSON around and Docker gets to (figuratively) tarball a bunch of `.so` files so that `ldd` can "dynamically" link them. The thing is, this way lies irrelevance. Even the GNU userland is pretty optional these days. Is Alpine, as I've taken to calling it, "GNU plus Linux"? Is NixOS? What are we putting up with this shit for? Bash?
- matheusmoreira 4y ago
  The history of free software is the history of the GPL and the power struggles around it. Free software developers neglecting binary interfaces in favor of source code, refusing to support plugins because it enables proprietary applications, new compilers arising to challenge GCC as a result, hardware makers using cryptography to prevent users from running their own code... And now we have this "hybrid linking" thing: static image containing dynamically linked libraries.

benreesman 4y ago

There's a big subthread on this post about it: https://news.ycombinator.com/item?id=32459755 https://news.ycombinator.com/item?id=32459755. Hermetic builds (Bazel, Nix, etc.) are in some sense orthogonal to packaging and delivery via e.g. a Docker container. It's true that it takes great care to get bit-for-bit identical output from a C/C++ toolchain, and it's also true that very subtle things can send the compiler and linker into a tailspin of "now I inline this and now I don't have room to inline that and now other optimization passes do something different" and whoa, my program just got a lot faster or slower, yikes. I forget the title now, but there was a great Strange Loop talk about how if you do good statistics on repeated builds or even just stuff more or less into `argv` that the difference between `-O2` and `-O3` doesn't hold up (on either `gcc` or `clang`). But whether the final executable is delivered to the target system via Docker or `scp` doesn't change any of that. If you want identical builds the Bazel and Nix people have a high-investment/high-return sales pitch they'd like you to hear :). And it's also true (as another commenter pointed out) that it's at least a little bit reductionist of me to say "Docker gives you static linking". Docker does a zillion things, and for the people using the other stuff, more power to them if it's working well. I would contend that most of them (especially since QEMU emulating x86_64 on your Intel Mac or vice versa isn't really a great story anymore) are not the real selling point though: you already had `virtualenv` and `nvm` and `rustup` and just countless ways to isolate/reproduce everything except the contents of your friendly neighborhood `/usr/lib/...`. If the native compiled code underpinning all of this stuff had all it's symbols linked in, and your kernel is the same or newer, you could use a tarball (and I've worked places where we did just that)! `git` was probably the tool that really sold everyone on: I want a goddamned cryptographic hash of everything in the thing, otherwise I Don't Trust Like That. We don't accept mystery meat from our source code anymore, why we feel any differently about what actually runs is Stockholm's syndrome IMHO.