4 ms·

Dependency hell exists because people still think static linking was not a great idea. All this while Rust, go, and all the new kids on the block use only stat

by Shorel 4y ago

Dependency hell exists because people still think static linking was not a great idea. All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. If you can update a library, you can also update all other software, and usually from the same source.

dehrmann 4y ago
> Dependency hell exists because people still think static linking was not a great idea. Then someone decided to call it Docker.
- Shorel 4y ago
  Exactly, containers are static linking with many extra steps. And more memory and storage requirements.
  - 0xbadcafebee 4y ago
    If your app calls the tool yq, is it written for yq v3, or v4? v4 has breaking changes in its interface. Just statically linking your app won't deal with that, but a container with both your app and yq v3 will. What containers don't deal with is the interfaces between containers.
saagarjha 4y ago
> All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. No, it's more that they were designed by entities that distributed their software using static linking.
- pcwalton 4y ago
  In Rust's case it was popular demand. Old Rust used dynamic linking a lot more. But our users demanded static linking (Go's popularity was a big driver of this), so the decision was made to do static linking by default.
  - saagarjha 4y ago
    Do you think Rust would have made more progress on stabilizing its ABI if dynamic linking was pursued?
    - eddyb 4y ago
      No, at most we would've invested more in e.g. ensuring¹ you cannot accidentally (or intentionally) mix-and-match shared objects not built by the same build process, or in ways to reduce the cost² of safe ("full relro") dynamic linking. ¹ via forcing the dynamic linker to eagerly resolve some symbols, global constructors, etc. - see https://github.com/rust-lang/rust/issues/73917#issuecomment-1200406761 https://github.com/rust-lang/rust/issues/73917#issuecomment-... for some recent discussion ² such as exporting a single base symbol per dylib (with a hash of code/data contents in the symbol name, for integrity), and statically resolving imports (to constant offsets from that base symbol), when linking object files into dylibs/executables - because of how position-independent code indirects through the "GOT", in practice this would mean that the dynamic linker would only need to lookup one symbol per dependency .so instead of one per import (of which there can be hundreds or thousands) Also, "dynamic linking" in Rust was never really "dynamic" or about "drop-in replacement to avoid rebuilds", it was only about the storage reuse of "shared objects", with the "late-binding" semantics inherent in dynamic linking a negative, not a positive. For example, rustc, rustdoc, clippy, miri, etc. are all relatively small executables that link against librustc_driver-*.so - on a recent nightly that's 120MiB, with only static linking that'd be half a GiB of executables instead. The relationship is "one .so shared between N executables" not "one .so per library". Also, if we could remove runtime symbol lookup but keep the separate .so (like ² above), we totally would. --- At the same time, Rust's continued evolution depends on being able to (almost) "never ask for permission" when changing internals that were never guaranteed to be one thing or another. Rust recently fixed most platforms to use simpler and more performant locks based on futex(-like) APIs (see https://github.com/rust-lang/rust/issues/93740 https://github.com/rust-lang/rust/issues/93740 and PRs like https://github.com/rust-lang/rust/pull/95035 https://github.com/rust-lang/rust/pull/95035), and this meant that e.g. Mutex<T> has a 32-bit (atomic) integer where a pointer used to be. An even more aggressive change was decoupling Ipv{4,6}Addr/SocketAddrV{4,6} from the C types they used to wrap internally (https://github.com/rust-lang/rust/pull/78802 https://github.com/rust-lang/rust/pull/78802) - that one actually required library authors to fix their code in a few cases (that were very incorrectly reinterpreting the std types as the C ones, without this ever being officially supported or condoned). Imagine trying to do that in C++. The stories I keep hearing from people involved in the C++ standard are tragic, more and more otherwise-uncontroversial improvements are being held back by poorly-motivated stubbornness around ABI stability. Large-scale users of C++, like Google, have long since migrated from C++'s standard library to their own replacements (e.g. abseil) for many of their codebases, and I kept hearing Google were "forking C++" (over the frozen ABI debacle specifically), long before it came out as "Carbon". --- The technology to coherently³ avoid today's rebuild-the-world cost in compiled languages⁴ isn't here yet IMO. ³ as in, guarantee identical behavior to the full rebuild from scratch, without introducing inconsistencies that could cause logic errors, memory corruption, etc. ⁴ some/most C libraries of course being the biggest exception - and while it's possible to intentionally design libraries like this in any language that can do FFI, it's fundamentally antithetical to "zero-cost abstractions" and compiler optimizations - nowadays the latter are forced even on otherwise-patchable C dependencies via LTO, in the interest of performance What I'm referring to is incremental recompilation with the following properties: ... [oops, hit comment size limit! I'll post the rest separately]
cesarb 4y ago
> All this while Rust, go, and all the new kids on the block use only static linking. This is trivially disproven: $ ldd /usr/bin/rustc linux-vdso.so.1 (0x00007ffd3de9b000) librustc_driver-e4385bf403336f76.so => /lib64/librustc_driver-e4385bf403336f76.so (0x00007ff832a00000) libstd-d371b708c754b3bb.so => /lib64/libstd-d371b708c754b3bb.so (0x00007ff832886000) libc.so.6 => /lib64/libc.so.6 (0x00007ff832600000) libLLVM-14.so => /lib64/libLLVM-14.so (0x00007ff82be00000) [...] Not only does rustc dynamically link against rust's "std" library, but also parts of rustc are implemented in a dynamic library ("librustc_driver"). The only reason projects written in Rust (other than rustc itself) don't use dynamic linking to other Rust code yet, is that there still isn't a stable ABI for Rust code (for good reason: that allowed them to introduce not only reordering of struct fields and niche optimizations, but also passing and returning slices as a pair of registers instead of a pointer to a two-word structure in the stack). I fully expect that, as the language matures, a stable (possibly opt-in) ABI will be introduced, together with freezing some of its standard library (for instance, the contents of the Vec structure, and how its inline functions work), allowing for dynamic linking against Rust's "std" (and as a consequence, passing Rust types between separate dynamically linked modules compiled with different releases of Rust). (As an aside: back when C was the "new kid on the block", it also used only static linking, dynamic linking of C code came later.)
- Shorel 4y ago
  Don't be that misleading. That's the rust compiler. You should analyze an executable created by the rust compiler instead. AFAIK only glibc and libgcc are linked dynamically in a program written in rust, and there's a way to statically link these.
  - steveklabnik 4y ago
    The Rust compiler is an executable created by the Rust compiler.
    - Shorel 4y ago
      And one of the very few, if not the only one, that have to link with LLVM.

by Shorel 4y ago

dehrmann 4y ago

> Dependency hell exists because people still think static linking was not a great idea. Then someone decided to call it Docker.

Shorel 4y ago
Exactly, containers are static linking with many extra steps. And more memory and storage requirements.
- 0xbadcafebee 4y ago
  If your app calls the tool yq, is it written for yq v3, or v4? v4 has breaking changes in its interface. Just statically linking your app won't deal with that, but a container with both your app and yq v3 will. What containers don't deal with is the interfaces between containers.

saagarjha 4y ago

> All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. No, it's more that they were designed by entities that distributed their software using static linking.

pcwalton 4y ago
In Rust's case it was popular demand. Old Rust used dynamic linking a lot more. But our users demanded static linking (Go's popularity was a big driver of this), so the decision was made to do static linking by default.
- saagarjha 4y ago
  Do you think Rust would have made more progress on stabilizing its ABI if dynamic linking was pursued?
  - eddyb 4y ago
    No, at most we would've invested more in e.g. ensuring¹ you cannot accidentally (or intentionally) mix-and-match shared objects not built by the same build process, or in ways to reduce the cost² of safe ("full relro") dynamic linking. ¹ via forcing the dynamic linker to eagerly resolve some symbols, global constructors, etc. - see https://github.com/rust-lang/rust/issues/73917#issuecomment-1200406761 https://github.com/rust-lang/rust/issues/73917#issuecomment-... for some recent discussion ² such as exporting a single base symbol per dylib (with a hash of code/data contents in the symbol name, for integrity), and statically resolving imports (to constant offsets from that base symbol), when linking object files into dylibs/executables - because of how position-independent code indirects through the "GOT", in practice this would mean that the dynamic linker would only need to lookup one symbol per dependency .so instead of one per import (of which there can be hundreds or thousands) Also, "dynamic linking" in Rust was never really "dynamic" or about "drop-in replacement to avoid rebuilds", it was only about the storage reuse of "shared objects", with the "late-binding" semantics inherent in dynamic linking a negative, not a positive. For example, rustc, rustdoc, clippy, miri, etc. are all relatively small executables that link against librustc_driver-*.so - on a recent nightly that's 120MiB, with only static linking that'd be half a GiB of executables instead. The relationship is "one .so shared between N executables" not "one .so per library". Also, if we could remove runtime symbol lookup but keep the separate .so (like ² above), we totally would. --- At the same time, Rust's continued evolution depends on being able to (almost) "never ask for permission" when changing internals that were never guaranteed to be one thing or another. Rust recently fixed most platforms to use simpler and more performant locks based on futex(-like) APIs (see https://github.com/rust-lang/rust/issues/93740 https://github.com/rust-lang/rust/issues/93740 and PRs like https://github.com/rust-lang/rust/pull/95035 https://github.com/rust-lang/rust/pull/95035), and this meant that e.g. Mutex<T> has a 32-bit (atomic) integer where a pointer used to be. An even more aggressive change was decoupling Ipv{4,6}Addr/SocketAddrV{4,6} from the C types they used to wrap internally (https://github.com/rust-lang/rust/pull/78802 https://github.com/rust-lang/rust/pull/78802) - that one actually required library authors to fix their code in a few cases (that were very incorrectly reinterpreting the std types as the C ones, without this ever being officially supported or condoned). Imagine trying to do that in C++. The stories I keep hearing from people involved in the C++ standard are tragic, more and more otherwise-uncontroversial improvements are being held back by poorly-motivated stubbornness around ABI stability. Large-scale users of C++, like Google, have long since migrated from C++'s standard library to their own replacements (e.g. abseil) for many of their codebases, and I kept hearing Google were "forking C++" (over the frozen ABI debacle specifically), long before it came out as "Carbon". --- The technology to coherently³ avoid today's rebuild-the-world cost in compiled languages⁴ isn't here yet IMO. ³ as in, guarantee identical behavior to the full rebuild from scratch, without introducing inconsistencies that could cause logic errors, memory corruption, etc. ⁴ some/most C libraries of course being the biggest exception - and while it's possible to intentionally design libraries like this in any language that can do FFI, it's fundamentally antithetical to "zero-cost abstractions" and compiler optimizations - nowadays the latter are forced even on otherwise-patchable C dependencies via LTO, in the interest of performance What I'm referring to is incremental recompilation with the following properties: ... [oops, hit comment size limit! I'll post the rest separately]

cesarb 4y ago

> All this while Rust, go, and all the new kids on the block use only static linking. This is trivially disproven: $ ldd /usr/bin/rustc linux-vdso.so.1 (0x00007ffd3de9b000) librustc_driver-e4385bf403336f76.so => /lib64/librustc_driver-e4385bf403336f76.so (0x00007ff832a00000) libstd-d371b708c754b3bb.so => /lib64/libstd-d371b708c754b3bb.so (0x00007ff832886000) libc.so.6 => /lib64/libc.so.6 (0x00007ff832600000) libLLVM-14.so => /lib64/libLLVM-14.so (0x00007ff82be00000) [...] Not only does rustc dynamically link against rust's "std" library, but also parts of rustc are implemented in a dynamic library ("librustc_driver"). The only reason projects written in Rust (other than rustc itself) don't use dynamic linking to other Rust code yet, is that there still isn't a stable ABI for Rust code (for good reason: that allowed them to introduce not only reordering of struct fields and niche optimizations, but also passing and returning slices as a pair of registers instead of a pointer to a two-word structure in the stack). I fully expect that, as the language matures, a stable (possibly opt-in) ABI will be introduced, together with freezing some of its standard library (for instance, the contents of the Vec structure, and how its inline functions work), allowing for dynamic linking against Rust's "std" (and as a consequence, passing Rust types between separate dynamically linked modules compiled with different releases of Rust). (As an aside: back when C was the "new kid on the block", it also used only static linking, dynamic linking of C code came later.)

Shorel 4y ago
Don't be that misleading. That's the rust compiler. You should analyze an executable created by the rust compiler instead. AFAIK only glibc and libgcc are linked dynamically in a program written in rust, and there's a way to statically link these.
- steveklabnik 4y ago
  The Rust compiler is an executable created by the Rust compiler.
  - Shorel 4y ago
    And one of the very few, if not the only one, that have to link with LLVM.