3 ms·

> You can't use multiple versions of a dependency on a single system without completely separating both the build-time and run-time environments Sure you can.

by fuzzy2 5y ago

> You can't use multiple versions of a dependency on a single system without completely separating both the build-time and run-time environments Sure you can. It's what version numbers in library file names are for. Works for executable names, too. Until Python 2.7 died, you commonly had 2.7 and 3.x installed at the same time. But maybe you meant something different?

throwaway984393 5y ago
It's impossible at the package manager level. First, install python 2 and python 3. Next, install a script that calls "#!/usr/bin/python". How do you make the script work, without modifying the script, so that it will work with the specific version of Python it was designed against? You can't. You have to either modify the script to point at the specific executable it was designed for, or you have to modify the environment to point "/usr/bin/python" at either python2 or python3. The former, "modifying the script", is recompiling all applications to point all of the executables they run at a custom executable, like "/run/app/v2/bin/app". The latter, "changing the environment", requires that you specify the environment before you run the script (". /etc/python2.env ; /some/script.py"). But then you have to know that, right now, you want to run the script against Python 2. What if you install an upgraded version of the script that's built against Python 3? How do you know which script to run when? You have to have an interface and pick which one to use each time you run it. And every application that runs that script. You can, of course, install two versions of your script, and rename the scripts, so that you know script2.py runs against python2 and script3.py runs against python3. But what if you have another program that needs to call the script? How does it know which of these 2 custom names to run? Now you'll have to compile a new custom package for this new program, and rename it too, and hard-code that one to call the specific new custom script name that targets the specific Python version. And on, and on, and on, you will need to have duplicate packages for every possible dependency graph that could possibly exist of different versions of different programs. You would end up with 16,000 packages of this python script, each with a custom name and custom paths. Because some combination of dependent packages could lead to a recursive bug where the wrong app eventually calls the wrong script. And it also requires patching literally every single application that is ever packaged by Nix to be able to rename applications and libraries. Every C program that ever calls "wc" needs to be patched to run "wc" version 2.0.0, "wc" version 2.0.1, "wc" version 3.0.0, etc, for every version of wc that may exist, and for every version of wc that may call a different program which also has multiple versions. This is why the problem is unsolvable with any tooling whatsoever. There is no way to automatically determine the correct dependencies without modifying and packaging every application for every dependency tree that could possibly exist. The only solution is what containers give you: an interface that forces you to pick a version of an application with one dependency tree, and lets you figure out how to combine them (how to run containers together). The only way to solve it without either 16,000 variants of one package, or an interface to pick environments at exec time, is to fundamentally change how software is designed and run today.
- clhodapp 5y ago
  The cases that you're raising are all ones that the nix ecosystem has had to find solutions for, but they have generally done so. The first thing to note is that it's generally discouraged to use full paths like the one you gave for python in shebangs in loose scripts. Instead, it is encouraged to write e.g. /usr/bin/env python. In this case, the python from the current environment will be picked up "impurely" when the script is run. That said, it's pretty heavily encouraged to package anything that will be used seriously, since this is very easy with nix and gives access to better tools. For example, there is built in automation for recognizing shell shebangs and rewriting them, based on the dependencies expressed in the build description. As far as your case of programs shelling out to other programs: that can either happen "impurely" (which would work like that you describe, including the potential drawbacks) or "purely" (where a specific version is captured). In order to attain "purity", nix has automation for generating wrapper scripts which manipulate the PATH before delegating to some underlying executable.

by fuzzy2 5y ago

throwaway984393 5y ago

It's impossible at the package manager level. First, install python 2 and python 3. Next, install a script that calls "#!/usr/bin/python". How do you make the script work, without modifying the script, so that it will work with the specific version of Python it was designed against? You can't. You have to either modify the script to point at the specific executable it was designed for, or you have to modify the environment to point "/usr/bin/python" at either python2 or python3. The former, "modifying the script", is recompiling all applications to point all of the executables they run at a custom executable, like "/run/app/v2/bin/app". The latter, "changing the environment", requires that you specify the environment before you run the script (". /etc/python2.env ; /some/script.py"). But then you have to know that, right now, you want to run the script against Python 2. What if you install an upgraded version of the script that's built against Python 3? How do you know which script to run when? You have to have an interface and pick which one to use each time you run it. And every application that runs that script. You can, of course, install two versions of your script, and rename the scripts, so that you know script2.py runs against python2 and script3.py runs against python3. But what if you have another program that needs to call the script? How does it know which of these 2 custom names to run? Now you'll have to compile a new custom package for this new program, and rename it too, and hard-code that one to call the specific new custom script name that targets the specific Python version. And on, and on, and on, you will need to have duplicate packages for every possible dependency graph that could possibly exist of different versions of different programs. You would end up with 16,000 packages of this python script, each with a custom name and custom paths. Because some combination of dependent packages could lead to a recursive bug where the wrong app eventually calls the wrong script. And it also requires patching literally every single application that is ever packaged by Nix to be able to rename applications and libraries. Every C program that ever calls "wc" needs to be patched to run "wc" version 2.0.0, "wc" version 2.0.1, "wc" version 3.0.0, etc, for every version of wc that may exist, and for every version of wc that may call a different program which also has multiple versions. This is why the problem is unsolvable with any tooling whatsoever. There is no way to automatically determine the correct dependencies without modifying and packaging every application for every dependency tree that could possibly exist. The only solution is what containers give you: an interface that forces you to pick a version of an application with one dependency tree, and lets you figure out how to combine them (how to run containers together). The only way to solve it without either 16,000 variants of one package, or an interface to pick environments at exec time, is to fundamentally change how software is designed and run today.

clhodapp 5y ago
The cases that you're raising are all ones that the nix ecosystem has had to find solutions for, but they have generally done so. The first thing to note is that it's generally discouraged to use full paths like the one you gave for python in shebangs in loose scripts. Instead, it is encouraged to write e.g. /usr/bin/env python. In this case, the python from the current environment will be picked up "impurely" when the script is run. That said, it's pretty heavily encouraged to package anything that will be used seriously, since this is very easy with nix and gives access to better tools. For example, there is built in automation for recognizing shell shebangs and rewriting them, based on the dependencies expressed in the build description. As far as your case of programs shelling out to other programs: that can either happen "impurely" (which would work like that you describe, including the potential drawbacks) or "purely" (where a specific version is captured). In order to attain "purity", nix has automation for generating wrapper scripts which manipulate the PATH before delegating to some underlying executable.