4 ms·

NixOS and Guix do _not_ require static linking. They go even one step further and allow many versions of a dynamic object to exist at the same time and each sof

by Throwaway194902 5y ago

NixOS and Guix do _not_ require static linking. They go even one step further and allow many versions of a dynamic object to exist at the same time and each software on your system chooses, which version of the object to link against.

AshamedCaptain 5y ago
You cannot change the version used by a given binary unless you rebuilt that binary, effectively making it similar to static linking (you still get sharing of DSO files & pages, though). i.e. even if I have two versions of some low-level library installed, depending installed packages still hardcode which one of the two low-level libraries to use, and if I want to switch the system from one to the other, I have to rebuilt it (or at least all depending packages). Suppose I make an overlay of libc to add some functionality that I am debugging which changes the libc binary, albeit not the ABI. Can I still reuse the same packages? Can I still reuse someone else's binary cache ? Basically, can I do without having to rebuild the entire system (save for libc) ?
- JamesSwift 5y ago
  If you are fine with sidestepping the system as intended then nothing prevents you from forcefully replacing the dynamically-linked, shared libc in the nix store.
- watusername 5y ago
  Yes, there is system.replaceRuntimeDependencies [1] that does what you ask: It replaces dependencies recursively in derivation outputs through binary patching, creating _new_ store paths (so immutability is preserved). It does so by replacing all occurrences of the store paths of the original dependency (e.g., libc) with the new ones [2]. [1] https://search.nixos.org/options?channel=21.11&show=system.replaceRuntimeDependencies&from=0&size=50&sort=relevance&type=packages&query=replaceRuntimeDependencies https://search.nixos.org/options?channel=21.11&show=system.r... [2] https://github.com/NixOS/nixpkgs/blob/fad04722fc3d692e3511e58e337ec9fa627f5ba5/pkgs/build-support/replace-dependency.nix#L3-L19 https://github.com/NixOS/nixpkgs/blob/fad04722fc3d692e3511e5...
- nh2 5y ago
  It is correct to say that dynamic linking with rebuild-on-input-dependency-change (like NixOS does) is similar to static linking when it comes to rebuild behaviour. However, also remember that changing dynamic libraries behind executables back is a concept that only makes sense in the presence of ABI compatibility. This is predominantly a C concept (or at least popular in the C world), and much less so for other linked programming languages like C++, Haskell, Go, and so on. Thus Nix being a general-purpose build system takes the general route here, and builds can also use their dependencies during the build step (e.g. a packaged program's autoconf suite might check the version or a symbol provided by a library), which requires full rebuilds for reproducibility. (Nix is working on content-addressed instead of input-hash-haddressed builds, which might open the door for avoiding many rebuilds that do not affect build _output_.) That said, it might still be quite easy to achieve what you want: * If you want to iterate on a C library that's early in the dependency graph, in am ABI-compatible fashion, you coul duse LD_LIBRARY_PATH or LD_PRELOAD on it. * If you want to override libc in an ABI-compatible way, you can use `patchelf` with its `--set-interpreter` and `--set-rpath` flags to replace the libc an executable is linked against after the fact. For example, you can make an `.override` of a nix derivation that just `cp -r`'s the existing files, and then calls `patchelf` on them. I have used both these methods to work around glibc bugs that I patched out, avoiding even to have to recompile my own software at the top of the build dependency tree. Some more relevant links about replacing glibc specifically: * https://github.com/NixOS/nixpkgs/issues/50329 https://github.com/NixOS/nixpkgs/issues/50329 * https://github.com/NixOS/nixpkgs/issues/129595 https://github.com/NixOS/nixpkgs/issues/129595 If you want to replace things system-wide instead of for your own software, the `system.replaceRuntimeDependencies` mentioned by a sibling comment might be a good choice.
- MayeulC 5y ago
  Guix has graft for something similar to this: https://guix.gnu.org/manual/devel/en/html_node/Security-Updates.html#index-grafts https://guix.gnu.org/manual/devel/en/html_node/Security-Upda... It's one of the points where it differs from Nix.
  - nh2 5y ago
    Is this different from Nix's `system.replaceRuntimeDependencies` posted in the sibling comment?
    - MayeulC 5y ago
      I was hoping to get an answer to your question as well, but I've only used Guix, so I can't really tell. Guix grafts are used to distribute updates, and they're quite easy to use, that's all I can say.
- civodul 5y ago
  Grafts in Guix support this use case: you know you're providing an ABI-compatible package replacement and don't want to rebuild everything that depends on it: https://guix.gnu.org/en/blog/2020/grafts-continued/ https://guix.gnu.org/en/blog/2020/grafts-continued/ We use that for security updates, but also in other situations where we know we can take advantage of it such as the new `--tune` package transformation option, which tunes a package for a specific CPU: https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/ https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu... Similarly, as a user, you can "graft" a replacement straight from the command line using `--with-graft`: https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html https://guix.gnu.org/manual/devel/en/html_node/Package-Trans...

AshamedCaptain 5y ago

You cannot change the version used by a given binary unless you rebuilt that binary, effectively making it similar to static linking (you still get sharing of DSO files & pages, though). i.e. even if I have two versions of some low-level library installed, depending installed packages still hardcode which one of the two low-level libraries to use, and if I want to switch the system from one to the other, I have to rebuilt it (or at least all depending packages). Suppose I make an overlay of libc to add some functionality that I am debugging which changes the libc binary, albeit not the ABI. Can I still reuse the same packages? Can I still reuse someone else's binary cache ? Basically, can I do without having to rebuild the entire system (save for libc) ?

JamesSwift 5y ago
If you are fine with sidestepping the system as intended then nothing prevents you from forcefully replacing the dynamically-linked, shared libc in the nix store.
watusername 5y ago
Yes, there is system.replaceRuntimeDependencies [1] that does what you ask: It replaces dependencies recursively in derivation outputs through binary patching, creating _new_ store paths (so immutability is preserved). It does so by replacing all occurrences of the store paths of the original dependency (e.g., libc) with the new ones [2]. [1] https://search.nixos.org/options?channel=21.11&show=system.replaceRuntimeDependencies&from=0&size=50&sort=relevance&type=packages&query=replaceRuntimeDependencies https://search.nixos.org/options?channel=21.11&show=system.r... [2] https://github.com/NixOS/nixpkgs/blob/fad04722fc3d692e3511e58e337ec9fa627f5ba5/pkgs/build-support/replace-dependency.nix#L3-L19 https://github.com/NixOS/nixpkgs/blob/fad04722fc3d692e3511e5...
nh2 5y ago
It is correct to say that dynamic linking with rebuild-on-input-dependency-change (like NixOS does) is similar to static linking when it comes to rebuild behaviour. However, also remember that changing dynamic libraries behind executables back is a concept that only makes sense in the presence of ABI compatibility. This is predominantly a C concept (or at least popular in the C world), and much less so for other linked programming languages like C++, Haskell, Go, and so on. Thus Nix being a general-purpose build system takes the general route here, and builds can also use their dependencies during the build step (e.g. a packaged program's autoconf suite might check the version or a symbol provided by a library), which requires full rebuilds for reproducibility. (Nix is working on content-addressed instead of input-hash-haddressed builds, which might open the door for avoiding many rebuilds that do not affect build _output_.) That said, it might still be quite easy to achieve what you want: * If you want to iterate on a C library that's early in the dependency graph, in am ABI-compatible fashion, you coul duse LD_LIBRARY_PATH or LD_PRELOAD on it. * If you want to override libc in an ABI-compatible way, you can use `patchelf` with its `--set-interpreter` and `--set-rpath` flags to replace the libc an executable is linked against after the fact. For example, you can make an `.override` of a nix derivation that just `cp -r`'s the existing files, and then calls `patchelf` on them. I have used both these methods to work around glibc bugs that I patched out, avoiding even to have to recompile my own software at the top of the build dependency tree. Some more relevant links about replacing glibc specifically: * https://github.com/NixOS/nixpkgs/issues/50329 https://github.com/NixOS/nixpkgs/issues/50329 * https://github.com/NixOS/nixpkgs/issues/129595 https://github.com/NixOS/nixpkgs/issues/129595 If you want to replace things system-wide instead of for your own software, the `system.replaceRuntimeDependencies` mentioned by a sibling comment might be a good choice.
MayeulC 5y ago
Guix has graft for something similar to this: https://guix.gnu.org/manual/devel/en/html_node/Security-Updates.html#index-grafts https://guix.gnu.org/manual/devel/en/html_node/Security-Upda... It's one of the points where it differs from Nix.
- nh2 5y ago
  Is this different from Nix's `system.replaceRuntimeDependencies` posted in the sibling comment?
  - MayeulC 5y ago
    I was hoping to get an answer to your question as well, but I've only used Guix, so I can't really tell. Guix grafts are used to distribute updates, and they're quite easy to use, that's all I can say.
civodul 5y ago
Grafts in Guix support this use case: you know you're providing an ABI-compatible package replacement and don't want to rebuild everything that depends on it: https://guix.gnu.org/en/blog/2020/grafts-continued/ https://guix.gnu.org/en/blog/2020/grafts-continued/ We use that for security updates, but also in other situations where we know we can take advantage of it such as the new `--tune` package transformation option, which tunes a package for a specific CPU: https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/ https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu... Similarly, as a user, you can "graft" a replacement straight from the command line using `--with-graft`: https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html https://guix.gnu.org/manual/devel/en/html_node/Package-Trans...