12 ms·
Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even cr
by captainmuon 5y ago
Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv.
What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them to `python`.
If I would get to redesign package management (both for Linux distros and for languages), I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time (for Python: at import time). Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. Instead, make it easy to get bleeding edge versions into the repositories. They can be installed side by side and only picked up by the things that actually use them.
- hk1337 5y agoThat sounds similar to what I do in macOS. I hate installing homebrew to /usr/local so I started installing it to ~/.brew and I hate using the python from homebrew so I always use pyenv.
- rizkeyz 5y agoThis is the real legacy problem: Python comes from a world, where only one version of one packaged seemed the right way to do. I do not have a good idea, but other ecosystems evolved much more sane in the realm of packaging. While not ideal, Go has done a fairly good job - and the "module" operations are instant - which they should be.
- 1337shadow 5y agoOk but Go compiles statically, while you can do the same with pyinstaller, I don't think that's really comparable as we're talking about deployment right there.
- rizkeyz 5y agoStatic binaries are a different story. Go has dependencies as any other modern language and they had a bad story in the past and have a better story today. Sketch for python: Create a ~/.cache/python/packages directory. Manage all dependencies there. Make the python interpreter "package aware" so that required dependencies are read off a file from the current project (e.g. "py.mod") and adjust "system path" accordingly and transparently. Or something along those lines. No extra tool, a single location, an easy to explain workflow (add a py.mod file, add deps there with versions, etc). I'm just thinking out loud, but it does not need to be hard.
- 1337shadow 5y ago> Manage all dependencies there Are you saying "put the different version of every dependency you need in there if you have to"? Because I don't think package managers are ready for that, they usually like to have one version installed per package.
- coldtea 5y ago>Because I don't think package managers are ready for that, they usually like to have one version installed per package. That's on them. Other language package managers can do it...
- 1337shadow 5y agoExactly, the whole topic is about easing the process for Linux distro packagers right?
- goodpoint 5y agoGo is terrible for distributions to package due to the poor versioning of libraries.
- jatone 5y agogo is incredible for distribution. you're distributing binaries not a bunch of source files. now the build tooling still needs some work around versions but its only a minor problem generally as much as it annoys me personally.
- goodpoint 5y agoDespite the downvotes, the argument stands: linux distributions are having a hard time handle the amount of tiny libraries and the conflicts in versioning and many maintainers voiced their concerns in the past years. The HN bubble can be amazing sometimes.
- PeterisP 5y agoThe point echoed in this discussion multiple times is that distributions should not handle the tiny python libraries and attempt to solve the dependency version issues, but treat an application with all its dependencies included as a single package. If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested, repackaged and rereleased for the update. This would cut down on the number of packages to be maintained, as the vast majority of python libraries would be exclude from the direct packaging process.
- goodpoint 5y ago> If a dependency needs to be bumped a version for e.g. security purposes, then the app obviously wasn't tested with the new version (which didn't exist at the time) and needs to be retested The burden of updating multiple copies of the same library across many packages grows exponentially and is simply untenable for distributions. If you can find an army of volunteers to do that, distributions would love their contributions. This hasn't happened in the last 20 years. I'd love to be proven wrong.
- nerdponx 5y agoHomebrew does this very well with its "cellar" system. Every version of every package gets installed to its own root tree, eg `/usr/local/Cellar/python/3.9.7/`. The currently-active version is then symlinked into `/usr/local/opt/python` and from there into `/usr/local`. I believe Nix and Guix also work this way.
- toyg 5y agoI'll remember that "Homebrew does this very well" the next time I have to fix a bunch of shit because it has updated the currently-active version, or removed this or that bugfix release, as part of a general upgrade. After the third time this happened, I started using pyenv - which is another mountain of brokenness, I grant you, but at least I have some degree of control on what happens when.
- drunkpotato 5y agopyenv is pretty good for working around this problem. I've recently switched to asdf-vm which I like even more, since it handles versions for multiple languages and tools.
- nerdponx 5y agoI have been meaning to try out ASDF-VM. Currently my shell initialization script has at least 4 of these "version managers". While I don't really mind them (and they are mostly well-behaved), it might be nice to have something a bit more centralized.
- drunkpotato 5y agoIn addition to python version issues, I was also running into JVM and gradle version compatibility issues, which I was handling with jenv and some aliases that would swap JAVAHOME environment variable as needed. asdf-vm cleaned all that up in a very clean way, and I like the way you can set .tool-versions file for a project and share with other asdf-vm users.
- 5y ago
- pimterry 5y ago> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I think this requirement made sense when disk space was scarce. I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author. Nowadays, I think neither is generally true. Disk is plentiful, distro packages are almost always far more out of date than the software's original source, and allowing authors to ship software with specific pinned dependency versions reduces bugs caused by dependency changes and makes providing support for software (and especially reproducing end-user's issues) significantly easier. Isolating dependencies by application, with linking to avoid outright duplication of identical versions (a la pnpm's approach for JS: https://pnpm.io/ https://pnpm.io/) is the way to go I think. Honestly, it feels like the way it's already gone, and it's just that the distros are fighting it tooth & nail whilst that approach takes over regardless.
- kijin 5y agoIt's not just about disk space, though. Distros assume responsibility for fixing major bugs and security vulnerabilities in the packages they ship. Old versions often contain bugs and vulnerabilities that new versions don't. Distros have two choices here: either ship the new version and remove the old version, or backport the fix to the old version. Continuing to ship the old version without the fix is not an option -- even if you also ship the new version -- because some programs will inevitably use the old version and then the distro will be on the hook for any resulting hacks. Backporting every fix to every version that ever shipped is also not a realistic project. Here in the startup world we often forget that there's a whole other market where many people would gladly accept 3-year-old versions in exchange for a guarantee of security fixes for 5-10 years. Someone needs to cater to this market, and the (non-rolling) distros perform that thankless task because individual developers won't.
- scotty79 5y ago> Distros assume responsibility for fixing major bugs and security vulnerabilities in the packages they ship. I think they should just ship Python programs, not libraries. They could check what libraries given Python program uses are safe in the version that it uses them. And just don't care if each of Python programs has a separate copy of the libraries or if particular version of particular library is shared between Python programs by Python environment. Distributions might just give up responsibility for sharing Python packages between Python programs without giving up the responsibility for security of those programs.
- BiteCode_dev 5y agoYes, explaining to a beginer how to bootstrap python is hell because windows, mac and linux keeps making sure nothing work the same.
- simonh 5y agoActually, explaining how to get Python working on Windows is far, far easier than on either linux (modulo various distros) or the Mac. That's because there is one obvious distribution of Python to use, the official one, and new versions of it are always consistent and play well together. Yes you can use Anaconda if you want, and people who do that are probably data scientists or something and know what they want to do and why. It's well documented and has it's own robust ecosystem. I say this as someone who's been on Macs at home since 2007 and works professionally on Linux, but I started with Python on Windows back in 2002.
- BiteCode_dev 5y agoUnfortunatly windows has plenty of problems too. First, the system PATH will kick you if of you have more than one python installed, so the official installer does not add to it by default, hence the python command doesn't work after an install. Instead, you get the py launcher, but it's not provided if you installed python from the app store or with anaconda. The you have powershell permission system. It's a ton of fun.
- toyg 5y ago> That's because there is one obvious distribution of Python to use Lol, no. There is one in the Windows appstore too, which has issues with privileges. And I'm almost sure now MS ships it in some other tool too.
- justin66 5y agoI wonder what percentage of the six people who use Windows app store have found that situation confusing.
- 5y ago
- cassianoleal 5y ago> I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time Essentially what Bundler has done for Ruby since pretty much forever. I would love to see this pattern elsewhere.
- yrro 5y ago> They even create a fake venv command that tells you to please install python-venv. ??? $ apt-file search /usr/bin/venv | wc -l 0 I do agree with you however that python3.9 should be moved into libpython3.9-stdlib, and hence be available to all installations.
- pydry 5y agoThey could have a separate environment for "system" python which excludes pip and is deliberately obscure (i.e. not /usr/bin/python).
- 1337shadow 5y agoInteresting idea, we should be able to hook before `sys.modules` cache or make on of such cache for each module in Python then we should be able to produce this. However, I thought the point was helping distro package management, which, to my knowledge, is not really built to support multiple installed versions of a package at the time: `dnf upgrade` for example, will upgrade all single instances of each packages to their newer release.
- 1337shadow 5y agoActually you can already override __import__ and implement this, but then you still need an installer, and distro support for multiple instances of a same package.
- 3np 5y agoThe problem arises when non-Python packages depend on Python modules. > If I would get to redesign package management (both for Linux distros and for languages), I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time (for Python: at import time). Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. Instead, make it easy to get bleeding edge versions into the repositories. They can be installed side by side and only picked up by the things that actually use them. You may want to check out Guix and Nix - their approach is pretty close to what you're describing. A common solution to this is if you still want to run traditional distros is to just run "bare infra" (whatever that means) on the host OS and everything else in containers or Nix.
- hnarn 5y ago> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. I’m not exactly sure how it works but I think I’ve heard that newer releases of Enterprise Linux (EL8+) support multiple channels of the same package or something similar.
- eqvinox 5y agoSorry, but to be frank, I think you're a bit ignorant here. Let me explain, starting from the bottom: > I would have one package manager that installs everything into a global package cache, […] There is exactly one package manager. If you're on Debian or Ubuntu, it's dpkg. If you're on RedHat, it's rpm. If you're on Arch, it's pacman. Yes, some of the BSDs have two (base packages + ports tree), but they're the odd ones out. pip, cargo, go*, etc. are not the same thing. I know they're called that but they don't perform the same function: none of them can create a working system installation. Let's call them module managers to have a distinct label. > and then pick the correct version of libraries at run time (for Python: at import time). That's easy for Python, and incredibly hard for a whole lot of other things. A module manager can do that. A package manager needs to work for a variety of code and ecosystems. Could it try to do it where possible? Maybe. But then the behavior is not uniform and made harder for users to understand. Could it still be worth it? Sure. But not obviously so. I would also say that this is just giving up on trying to keep a reasonable ecosystem. It's not impossible to reduce the dependency hell that some things have devolved into. It just needs interest in doing so, and discipline while effecting it. I'd really prefer not giving up on this. > Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. This is to some degree why distros are breaking apart Python. Some bits are easy to install in parallel, some aren't. There can only be one "python". Worse, there can only be one "libpython3.9.so.1.0". > Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. They're trying to achieve the very goals you're describing. Trying to give you a working python without having to download and "install" some weird thing somewhere else. And at the same time trying to keep the module managers working when they're replacing some module but not all of them. On a subjective level, it's obvious you have a strong distaste for this ("they even create a fake") — but could you please make objective arguments how and why this breaks things? If you're getting an incomplete Python installation, that seems like a packaging bug the distro needs to fix. Is that it? Or are there other issues? > If I would get to redesign package management (both for Linux distros and for languages), Well, and now we're here: https://xkcd.com/927/ https://xkcd.com/927/ And, I'm sorry to say this, but your post does not convey to me the existence of any essential C codebase packaging knowledge on your end. I don't know about other ecosystems, but I have done packaging work on C codebases (with Python bindings no less), and you don't seem to be aware of very basic issues like header/library mismatches and runtime portability. If you are interested in this topic, please familiarize yourself with the world of existing package managers, the problems they run into, and how they solve them. There's a lot to learn there, and they're quite distinct from each other on some fronts too. Some problems are still unsolved even.
- thereddaikon 5y agoThis behavior is why things like Snaps and Flatpacks have become so popular. Package managers operate under a draconian and outdated mindset that gets in the way more than it helps at this stage. You can both allow different versions of the same packages to coexist while also managing updates and installation/removal of software. It doesn't have to be this way. Software should be able to ship with its dependencies included and work and not rely on the whims of the OS getting it right.
- stonecharioteer 5y agoCheckout the deadsnakes apt repository. It contains all versions of python you could need, and you can install them side by side.
- TingPing 5y agoFedora works similar to this. They just package them all and you use which ever one for your private needs.