8 ms·

Nix is running face-first into complexities of build and package managers. As an observer it looks like Python's package ecosystem in particular is a giant mess

Nix is running face-first into complexities of build and package managers. As an observer it looks like Python's package ecosystem in particular is a giant mess. This affects Nix disproportionately because Nix actually integrates all package updates into one channel which nobody else anywhere does (and I guess package authors often don't care to fix), and it's even worse because a lot of projects use python as a build dependency (?!) which then cascades these issues even farther. I've been following this thread [1] about the issue which has valid interested parties including users, package authors, and nix package maintainers, and contains various proposals to solve or alleviate the problems. [1]: https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741 https://discourse.nixos.org/t/nixpkgss-current-development-w...

pxc 4y ago
Yeah. This is covering stuff you know if you've read that thread, but: Putting all Python libs into a single channel is something Nixpkgs does because Python can't handle different versions of the same library in a single process. The Python libs that are used in actual applications in NixOS, then, need to be compatible or they might cause weird issues when a new or existing package tries to leverage them at the same time. Other distros do run into this but it might be worse in Nix. (Some (all?) C libs have this same issue, but they don't have rampant integration problems from version to version like the Python ecosystem does, so having just one copy of them is fine.) And yeah, Python developers seem wary of the kind of vendorization that would fix this, and some Python package authors are very hostile about integration issues that Linux distros are more likely to hit than developers of individual downstream applications are. (I guess they don't feel like distro integrators are truly 'using their code' in the way that application or library developers would be, and those integration issues can be a lot of work to figure out for what feels like some 'fake user's' configuration problem.) Python packaging has been an incredible mess for many, many years. I don't think the community's processes and institutions have the means to meaningfully fix it so that downstream consumers of Python packaging infrastructure don't have to use a big pile of hacks to successfully package Python applications. Distros will probably see another language replace Python for internal tooling and sysadmin applications before they see Python packaging unified around something that behaves deterministically, works offline, lets programs reason about the dependencies of packages that are not present/installed/built, pins versions with cryptographic hashes by default, disallows arbitrary scripts at install time, sanely describes dependencies on native libraries, etc. PS: This is not a knock on Python. The Python community faces some really tough institutional/governance/adoption problems because: - the language is mature and the ecosystem has a lot of valuable code in it, already packaged in various ways - the language was born without a modern package management story, because it's very old! - big community-wide changes are democratically governed and there are a lot of stakeholders who are bound to have opinions Consensus will be really hard to build and legacy packaging processes will stick around for a long time. And some changes that could really help, like in-process changes to library loading behavior, are likely to be seen (perhaps correctly!) as too radical/disruptive to be in the best interest of the majority of the existing community.
- jnxx 4y ago
  > The Python community faces some really tough institutional/governance/adoption problems The Linux kernel people could have the same problems... it is genius how they have solved them, by developing git but also by establishing processes that scale.
  - qbasic_forever 4y ago
    In the Linux kernel world Linus is the final arbiter and decision-maker--he takes input from people he trusts but ultimately he and only he has the final commit. Python used to have this model too with Guido at the top as the "BDFL" (benevolent dictator for life), however when he stepped down they replaced it with a consensus of a steering committee: https://peps.python.org/pep-0013/ https://peps.python.org/pep-0013/ Both projects have very different governance as a result.
- SuperSandro2000 4y ago
  Applications can overwrite dependency versions they want to use. It's just not possible in the python package set because if ever two versions of the same package are in one environment very strange things happen and everything falls apart.
  - pxc 4y ago
    Yep, Nixpkgs freely relies on that kind of vendorization to solve compatibility problems in other programming languages' ecosystems when it has to. And Nix users can totally override specific Python library versions as one-offs for their own Python packages or environments. It's just really unfortunate that some of Python's internals make that unfeasible for Nixpkgs' collection of Python libraries as a whole. :(
jnxx 4y ago
> This affects Nix disproportionately because Nix actually integrates all package updates into one channel which nobody else anywhere does Does Arch not do that?
- infogulch 4y ago
  Maybe it would be more correct to say that "Nix integrates all package updates into one channel which is unique to system integrators (like Nix and Arch) but an uncommon workflow for regular application developers".
  - pxc 4y ago
    Yep, I think that revision is perfect.
jnxx 4y ago
> As an observer it looks like Python's package ecosystem in particular is a giant mess. Most people would say it is. The better question is how to avoid such a mess. My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. And yeah, that has also become a cultural issue with Python, as the Python2/3 breakage shows. So, could one sum it up in that Nix magnifies unsolved compatibility issues in packaging systems? Because if there were a single core Python distribution, like say, Anaconda, but nothing else, these issues would not exist. Of course, people can avoid the issues if they only use a handful of packages. But putting all into a single channel makes the problem much more acute.
- infogulch 4y ago
  > Nix magnifies unsolved compatibility issues in packaging systems ... putting all into a single channel makes the problem much more acute Well said.
- lamontcg 4y ago
  > My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. Given that software developers never guess the correct design up front this means that you always have architecturally buggy software, and a bunch of complaining about why buggy-looking edge conditions are never fixed. There has to be some kind of release valve for software to evolve and break backwards compatibility.
  - infogulch 4y ago
    Yes they've invented that: It's called bumping the major version.
    - lamontcg 4y ago
      The parent comment is complaining about Python 2/3 breakage which was a major version.
SuperSandro2000 4y ago
The python ecosystem is not a giant mess, its just dependency hell. You need to work with what you get from upstream. > Nix actually integrates all package updates into one channel which nobody else anywhere does If you are using the python system packages then Arch, Debian and probably more are doing the same. > it's even worse because a lot of projects use python as a build dependency (?!) which then cascades these issues even farther. In practice this is not a problem at all. Build systems that use python underneath have usually very little dependencies. The bigger problems we are facing are big python (web) applications and anything that moves (very) slowly upstream like (sadly) many AI/ML projects. FYI: I am very active NixOS maintainer and one of my foci is the python packages in NixOS.
- infogulch 4y ago
  Thank you for your input! What do you think about the linked thread? > The python ecosystem is not a giant mess, its just dependency hell. There are dozens (?) of actively used package and environment management systems, with no consistency and no lockfiles, many packages ignore semver, and having multiple versions of a package installed causes weird issues. I'm not sure why "a giant mess" is an invalid descriptor, I guess it's just arguing semantics.
  - baby 4y ago
    I’m guessing you’d just have to create a derivation for the python package you’re trying to use.
  - viraptor 4y ago
    Semver doesn't really apply here since nix always points at a specific version of each dependency. Multiple package versions are not visible to the same app. And you can include your own lockfile-like list. So in practice, it's not really a bigger problem for nix than for any other release. If you need to freeze versions for an app, you freeze them. If you don't or if you're packaging a library, you can (usually) rely on automated PR testing to catch breakages and deal with them there.
    - 4y ago
JonChesterfield 4y ago
Python is _really_ tempting as a build (or test) dependency. It's an easy way to throw together code generators or similar glue that is likely to run on whatever platform is building your software.

pxc 4y ago

Yeah. This is covering stuff you know if you've read that thread, but: Putting all Python libs into a single channel is something Nixpkgs does because Python can't handle different versions of the same library in a single process. The Python libs that are used in actual applications in NixOS, then, need to be compatible or they might cause weird issues when a new or existing package tries to leverage them at the same time. Other distros do run into this but it might be worse in Nix. (Some (all?) C libs have this same issue, but they don't have rampant integration problems from version to version like the Python ecosystem does, so having just one copy of them is fine.) And yeah, Python developers seem wary of the kind of vendorization that would fix this, and some Python package authors are very hostile about integration issues that Linux distros are more likely to hit than developers of individual downstream applications are. (I guess they don't feel like distro integrators are truly 'using their code' in the way that application or library developers would be, and those integration issues can be a lot of work to figure out for what feels like some 'fake user's' configuration problem.) Python packaging has been an incredible mess for many, many years. I don't think the community's processes and institutions have the means to meaningfully fix it so that downstream consumers of Python packaging infrastructure don't have to use a big pile of hacks to successfully package Python applications. Distros will probably see another language replace Python for internal tooling and sysadmin applications before they see Python packaging unified around something that behaves deterministically, works offline, lets programs reason about the dependencies of packages that are not present/installed/built, pins versions with cryptographic hashes by default, disallows arbitrary scripts at install time, sanely describes dependencies on native libraries, etc. PS: This is not a knock on Python. The Python community faces some really tough institutional/governance/adoption problems because: - the language is mature and the ecosystem has a lot of valuable code in it, already packaged in various ways - the language was born without a modern package management story, because it's very old! - big community-wide changes are democratically governed and there are a lot of stakeholders who are bound to have opinions Consensus will be really hard to build and legacy packaging processes will stick around for a long time. And some changes that could really help, like in-process changes to library loading behavior, are likely to be seen (perhaps correctly!) as too radical/disruptive to be in the best interest of the majority of the existing community.

jnxx 4y ago
> The Python community faces some really tough institutional/governance/adoption problems The Linux kernel people could have the same problems... it is genius how they have solved them, by developing git but also by establishing processes that scale.
- qbasic_forever 4y ago
  In the Linux kernel world Linus is the final arbiter and decision-maker--he takes input from people he trusts but ultimately he and only he has the final commit. Python used to have this model too with Guido at the top as the "BDFL" (benevolent dictator for life), however when he stepped down they replaced it with a consensus of a steering committee: https://peps.python.org/pep-0013/ https://peps.python.org/pep-0013/ Both projects have very different governance as a result.
SuperSandro2000 4y ago
Applications can overwrite dependency versions they want to use. It's just not possible in the python package set because if ever two versions of the same package are in one environment very strange things happen and everything falls apart.
- pxc 4y ago
  Yep, Nixpkgs freely relies on that kind of vendorization to solve compatibility problems in other programming languages' ecosystems when it has to. And Nix users can totally override specific Python library versions as one-offs for their own Python packages or environments. It's just really unfortunate that some of Python's internals make that unfeasible for Nixpkgs' collection of Python libraries as a whole. :(

jnxx 4y ago

> This affects Nix disproportionately because Nix actually integrates all package updates into one channel which nobody else anywhere does Does Arch not do that?

infogulch 4y ago
Maybe it would be more correct to say that "Nix integrates all package updates into one channel which is unique to system integrators (like Nix and Arch) but an uncommon workflow for regular application developers".
- pxc 4y ago
  Yep, I think that revision is perfect.

jnxx 4y ago

> As an observer it looks like Python's package ecosystem in particular is a giant mess. Most people would say it is. The better question is how to avoid such a mess. My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. And yeah, that has also become a cultural issue with Python, as the Python2/3 breakage shows. So, could one sum it up in that Nix magnifies unsolved compatibility issues in packaging systems? Because if there were a single core Python distribution, like say, Anaconda, but nothing else, these issues would not exist. Of course, people can avoid the issues if they only use a handful of packages. But putting all into a single channel makes the problem much more acute.

infogulch 4y ago
> Nix magnifies unsolved compatibility issues in packaging systems ... putting all into a single channel makes the problem much more acute Well said.
lamontcg 4y ago
> My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. Given that software developers never guess the correct design up front this means that you always have architecturally buggy software, and a bunch of complaining about why buggy-looking edge conditions are never fixed. There has to be some kind of release valve for software to evolve and break backwards compatibility.
- infogulch 4y ago
  Yes they've invented that: It's called bumping the major version.
  - lamontcg 4y ago
    The parent comment is complaining about Python 2/3 breakage which was a major version.

SuperSandro2000 4y ago

The python ecosystem is not a giant mess, its just dependency hell. You need to work with what you get from upstream. > Nix actually integrates all package updates into one channel which nobody else anywhere does If you are using the python system packages then Arch, Debian and probably more are doing the same. > it's even worse because a lot of projects use python as a build dependency (?!) which then cascades these issues even farther. In practice this is not a problem at all. Build systems that use python underneath have usually very little dependencies. The bigger problems we are facing are big python (web) applications and anything that moves (very) slowly upstream like (sadly) many AI/ML projects. FYI: I am very active NixOS maintainer and one of my foci is the python packages in NixOS.

infogulch 4y ago
Thank you for your input! What do you think about the linked thread? > The python ecosystem is not a giant mess, its just dependency hell. There are dozens (?) of actively used package and environment management systems, with no consistency and no lockfiles, many packages ignore semver, and having multiple versions of a package installed causes weird issues. I'm not sure why "a giant mess" is an invalid descriptor, I guess it's just arguing semantics.
- baby 4y ago
  I’m guessing you’d just have to create a derivation for the python package you’re trying to use.
- viraptor 4y ago
  Semver doesn't really apply here since nix always points at a specific version of each dependency. Multiple package versions are not visible to the same app. And you can include your own lockfile-like list. So in practice, it's not really a bigger problem for nix than for any other release. If you need to freeze versions for an app, you freeze them. If you don't or if you're packaging a library, you can (usually) rely on automated PR testing to catch breakages and deal with them there.
  - 4y ago

JonChesterfield 4y ago

Python is _really_ tempting as a build (or test) dependency. It's an easy way to throw together code generators or similar glue that is likely to run on whatever platform is building your software.