3 ms·
Yeah. This is covering stuff you know if you've read that thread, but: Putting all Python libs into a single channel is something Nixpkgs does because Python c
by pxc 4y ago
Yeah. This is covering stuff you know if you've read that thread, but:
Putting all Python libs into a single channel is something Nixpkgs does because Python can't handle different versions of the same library in a single process. The Python libs that are used in actual applications in NixOS, then, need to be compatible or they might cause weird issues when a new or existing package tries to leverage them at the same time. Other distros do run into this but it might be worse in Nix. (Some (all?) C libs have this same issue, but they don't have rampant integration problems from version to version like the Python ecosystem does, so having just one copy of them is fine.)
And yeah, Python developers seem wary of the kind of vendorization that would fix this, and some Python package authors are very hostile about integration issues that Linux distros are more likely to hit than developers of individual downstream applications are. (I guess they don't feel like distro integrators are truly 'using their code' in the way that application or library developers would be, and those integration issues can be a lot of work to figure out for what feels like some 'fake user's' configuration problem.)
Python packaging has been an incredible mess for many, many years. I don't think the community's processes and institutions have the means to meaningfully fix it so that downstream consumers of Python packaging infrastructure don't have to use a big pile of hacks to successfully package Python applications.
Distros will probably see another language replace Python for internal tooling and sysadmin applications before they see Python packaging unified around something that behaves deterministically, works offline, lets programs reason about the dependencies of packages that are not present/installed/built, pins versions with cryptographic hashes by default, disallows arbitrary scripts at install time, sanely describes dependencies on native libraries, etc.
PS: This is not a knock on Python. The Python community faces some really tough institutional/governance/adoption problems because:
- the language is mature and the ecosystem has a lot of valuable code in it, already packaged in various ways
- the language was born without a modern package management story, because it's very old!
- big community-wide changes are democratically governed and there are a lot of stakeholders who are bound to have opinions
Consensus will be really hard to build and legacy packaging processes will stick around for a long time. And some changes that could really help, like in-process changes to library loading behavior, are likely to be seen (perhaps correctly!) as too radical/disruptive to be in the best interest of the majority of the existing community.
- jnxx 4y ago> The Python community faces some really tough institutional/governance/adoption problems The Linux kernel people could have the same problems... it is genius how they have solved them, by developing git but also by establishing processes that scale.
- qbasic_forever 4y agoIn the Linux kernel world Linus is the final arbiter and decision-maker--he takes input from people he trusts but ultimately he and only he has the final commit. Python used to have this model too with Guido at the top as the "BDFL" (benevolent dictator for life), however when he stepped down they replaced it with a consensus of a steering committee: https://peps.python.org/pep-0013/ https://peps.python.org/pep-0013/ Both projects have very different governance as a result.
- SuperSandro2000 4y agoApplications can overwrite dependency versions they want to use. It's just not possible in the python package set because if ever two versions of the same package are in one environment very strange things happen and everything falls apart.
- pxc 4y agoYep, Nixpkgs freely relies on that kind of vendorization to solve compatibility problems in other programming languages' ecosystems when it has to. And Nix users can totally override specific Python library versions as one-offs for their own Python packages or environments. It's just really unfortunate that some of Python's internals make that unfeasible for Nixpkgs' collection of Python libraries as a whole. :(