5 ms·

> where multiple versions of the same library or application can live in harmony How does NixOS solve the diamond dependency problem?

geofft 10y ago
I think in practice, I have far less trouble with the actual diamond dependency problem - binary A requires libraries B and C, which depend on two different versions of D - than the boring version - system A runs binaries B and C, which depend on two different versions of D. NixOS solves the latter.
corysama 10y ago
If you are asking about a single application depending on conflicting libraries such that it can't be built at all, I don't think NixOS can solve that. What it can do is have multiple conflicting libraries installed simultaneously and individual applications can choose which completely-defined environment they want to run in.
smallnamespace 10y ago
It's largely mitigated by allowing multiple versions. Imagine A depends on B and C, which both depend on D. The diamond dependency usually pops up when B and C need different versions of D. But in this case, Nix allows you to simply install another version of B or C such that they can share D. This doesn't help you if there's no set of A ... D that are jointly consistent, but then you would've been SOL anyway.
ambrop7 10y ago
Normally you end up only a single version of a library needed in the scope of one application. E.g. application A depends on libraries B and C and both of these depend on the same version of library D. And there is no problem as long as this is the case, the exact same library will be used. If for some reason B and C depend on different versions of a library, then things may go wrong, e.g. one package may end up inadvertently using an incorrect version of a dependency. Nix cannot solve this problem because it originates from the OS/compiler design (same sonames, same symbol names). Consider that with respect to libraries, Nix mostly just wraps the compiler to add needed -L flags and also sets RPATH in shared libs and executables so that needed shared libs are found at runtime. If an application effectively depends on different versions of one library with the same soname, the dynamic linker will still use just one of the available ones. What Nix does allow you to do is to isolate packages from one another. You can have one application using only one version of library D, and another application using only a different version of library D. I should note that it's somehow hard to get to this situation, because the Nix packages collection (nixpkgs) doesn't keep too many versions of packages. Typically for minor upgrades the library is just bumped, and when you rebuild the final application all references to that library will be up to date. You can only realistically get issues if the application ends up depending on different major versions which are both maintained in nixpkgs but are not designed to work together in the same application.
- nine_k 10y ago
  I suspect that the same binary cannot realistically depend on two different versions of a library. I can imagine that package A may have two binaries, one depending on a library B version 1.1, and another, on B 1.2 Does NixOS allow to have both B-1.1 and B-1.2 installed, and A's different binaries to load the correct different versions of the library from B-1.1 and B-1.2?
  - ambrop7 10y ago
    Well yes strictly speaking, the same binary. But in practice with Nix's default library resolution logic (the mentioned compiler wrapping) you will probably also have problems trying to build binaries in one package using version of a library (see note [1]). That is, I think you'd have to do some "hacking" to fix the issue, or separate the binaries into different packages (and maybe create a new package that "assembles" all the binaries by creating symlinks to them). > Does NixOS allow to have both B-1.1 and B-1.2 installed, and A's different binaries to load the correct different versions of the library from B-1.1 and B-1.2? Yes, well actually, the thing is that typically you only "install" (into a user environment) final applications (A) that you need not libraries, so the libraries (B) can be considered to not really be installed, just available (somewhere in /nix/store) and referenced correctly (e.g. by RPATH). [1] When you declare a dependency in a package specification by putting it into buildInputs, the compiler wrapper will automatically add an -L<store_path>/lib so that linking with just -l will find the library. With multiple versions of a library in buildInputs, that becomes a problem.
  - dbcurtis 10y ago
    As I understand it, Nixos runs their own patched versions of ld and the dynamic loading system calls in libc. The normal lib.so search mechanism is replaced wholesale with "thou shalt link against this dll and no other" on a per-execuatble basis. Multiple versions of .so's don't cause problems, because no executable is depending on a search mechanism it doesn't control.
    - ambrop7 10y ago
      I think that they don't do too drastic changes. There are certainly checks in the compilation process to not accidentally link to /usr/lib/ libraries, but I think the main problem of finding libraries works with a simple compiler wrapper adding needed -Lflags (for finding libs at compile time) and RPATH entries to binaries (for finding libs at run time). There of course may be "hacks" made to gcc/libc but I don't see a reason they would be essential for functioning. One notable thing Nix does is hardcode a path to a specific dynamic linker (e.g. /nix/store/HASH-glibc-2.23/lib/ld-linux-x86-64.so.2). All other dynamic libraries should be referenced through soname+RPATH.

geofft 10y ago

I think in practice, I have far less trouble with the actual diamond dependency problem - binary A requires libraries B and C, which depend on two different versions of D - than the boring version - system A runs binaries B and C, which depend on two different versions of D. NixOS solves the latter.

corysama 10y ago

If you are asking about a single application depending on conflicting libraries such that it can't be built at all, I don't think NixOS can solve that. What it can do is have multiple conflicting libraries installed simultaneously and individual applications can choose which completely-defined environment they want to run in.

smallnamespace 10y ago

It's largely mitigated by allowing multiple versions. Imagine A depends on B and C, which both depend on D. The diamond dependency usually pops up when B and C need different versions of D. But in this case, Nix allows you to simply install another version of B or C such that they can share D. This doesn't help you if there's no set of A ... D that are jointly consistent, but then you would've been SOL anyway.

ambrop7 10y ago

Normally you end up only a single version of a library needed in the scope of one application. E.g. application A depends on libraries B and C and both of these depend on the same version of library D. And there is no problem as long as this is the case, the exact same library will be used. If for some reason B and C depend on different versions of a library, then things may go wrong, e.g. one package may end up inadvertently using an incorrect version of a dependency. Nix cannot solve this problem because it originates from the OS/compiler design (same sonames, same symbol names). Consider that with respect to libraries, Nix mostly just wraps the compiler to add needed -L flags and also sets RPATH in shared libs and executables so that needed shared libs are found at runtime. If an application effectively depends on different versions of one library with the same soname, the dynamic linker will still use just one of the available ones. What Nix does allow you to do is to isolate packages from one another. You can have one application using only one version of library D, and another application using only a different version of library D. I should note that it's somehow hard to get to this situation, because the Nix packages collection (nixpkgs) doesn't keep too many versions of packages. Typically for minor upgrades the library is just bumped, and when you rebuild the final application all references to that library will be up to date. You can only realistically get issues if the application ends up depending on different major versions which are both maintained in nixpkgs but are not designed to work together in the same application.

nine_k 10y ago
I suspect that the same binary cannot realistically depend on two different versions of a library. I can imagine that package A may have two binaries, one depending on a library B version 1.1, and another, on B 1.2 Does NixOS allow to have both B-1.1 and B-1.2 installed, and A's different binaries to load the correct different versions of the library from B-1.1 and B-1.2?
- ambrop7 10y ago
  Well yes strictly speaking, the same binary. But in practice with Nix's default library resolution logic (the mentioned compiler wrapping) you will probably also have problems trying to build binaries in one package using version of a library (see note [1]). That is, I think you'd have to do some "hacking" to fix the issue, or separate the binaries into different packages (and maybe create a new package that "assembles" all the binaries by creating symlinks to them). > Does NixOS allow to have both B-1.1 and B-1.2 installed, and A's different binaries to load the correct different versions of the library from B-1.1 and B-1.2? Yes, well actually, the thing is that typically you only "install" (into a user environment) final applications (A) that you need not libraries, so the libraries (B) can be considered to not really be installed, just available (somewhere in /nix/store) and referenced correctly (e.g. by RPATH). [1] When you declare a dependency in a package specification by putting it into buildInputs, the compiler wrapper will automatically add an -L<store_path>/lib so that linking with just -l will find the library. With multiple versions of a library in buildInputs, that becomes a problem.
- dbcurtis 10y ago
  As I understand it, Nixos runs their own patched versions of ld and the dynamic loading system calls in libc. The normal lib.so search mechanism is replaced wholesale with "thou shalt link against this dll and no other" on a per-execuatble basis. Multiple versions of .so's don't cause problems, because no executable is depending on a search mechanism it doesn't control.
  - ambrop7 10y ago
    I think that they don't do too drastic changes. There are certainly checks in the compilation process to not accidentally link to /usr/lib/ libraries, but I think the main problem of finding libraries works with a simple compiler wrapper adding needed -Lflags (for finding libs at compile time) and RPATH entries to binaries (for finding libs at run time). There of course may be "hacks" made to gcc/libc but I don't see a reason they would be essential for functioning. One notable thing Nix does is hardcode a path to a specific dynamic linker (e.g. /nix/store/HASH-glibc-2.23/lib/ld-linux-x86-64.so.2). All other dynamic libraries should be referenced through soname+RPATH.