5 ms·

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of th

Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, some need long term stability while others need bleeding edge packages that haven’t been released yet. Solving for a single case is straightforward; solving for all of them , not so much.

microtonal 5y ago
And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems, like NixOS or OSTree-based system like Silverblue), a wide variety of build systems that sometimes hook into make, etc. Solving this is a hard problem. This is why an authority like PSF has to step in and say: this is how it is going to be done from here onwards. [1] E.g. Rust's Cargo.
- komuher 5y ago
  Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-
  - smallerfish 5y ago
    Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.
    - RegW 5y ago
      Ant didn't do package management.
      - smallerfish 5y ago
        No, but ivy was commonly used. That said, the bigger point is that the standard way to do things can be changed. For some reason, despite (or because of?) PEP, the python community seems unable to coalesce around standard ways of packaging.
        RegW 5y ago
        Perhaps I have lived too long out in the provences, but Maven was my first experience of dependency management in Java. After Ant it was a no-brainer because Ant didn't do dependencies. I didn't come across references to Ivy for some years and personally I have never seen it used in the wild. That said, I didn't actually like early Maven that much, it was pretty inflexible and often required the AntRun plugin to do something novel. However, it is still top dog everywhere I work and very much the standard way to do things.
  - WhyNotHugo 5y ago
    Which is why I'm starting to feel like Python is the new Java.
    - oneweekwonder 5y ago
      Strange, because python was released almost 5 years before java.
      - mangecoeur 5y ago
        Python is the Old Java :P
  - microtonal 5y ago
    Which is exactly what my second paragraph said :). This is a common struggle for many older programming language ecosystems, e.g. I think the same is true to some extend for C and C++. As one of the sibling commenters mentions, there are good examples that it is possible to standardize packaging better. Maven replaced IDE-driven builds and Ant for in the Java ecosystem and added proper package management. Additionally, it required that projects start conforming to standardized layouts, by taking convention over configuration and being largely declarative. I think the Maven success story lost some of its shine with Gradle, but that's another story.
  - ajsnigrutin 5y ago
    Perl is 33 years old, and perl 5.x is 27 years old. I can literally take a 20+yo book and all the examples still work. CPAN still works. I literally have 20+yo scripts still copied from server to server, from laptop to laptop, without any changes.
- __float 5y ago
  Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version). Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, unsupported, but up to date".
  - wildbook 5y ago
    Rust almost does the exact same thing, so I'd say it's fair. Dependencies (crates) are grabbed in source format and compiled locally as part of the build process, and installing Rust programs through Cargo also compiles them (and their dependencies) locally. Some crates have the same issue where build scripts rely on outside tooling being installed, but it's definitely not common to (unless you're relying on compiling C/C++ code for FFI for example, in which case it's somewhat frequent).
    - __float 5y ago
      I think there needs to be a distinction between fetching crates as source at _build time_ and what happens with Python. Python's "build time" will still require the source to be present on the target machine it's deployed to -- unfortunately these tools are complex because "packaging" is only half of the issue, it's also the distribution and _deployment_ which makes things messy. Consider that, even if you want to use packages only from your application's virtualenv, the default (footgun warning!) is that Python will still use the "system" packages -- this means you may have installed Ansible or some other tool that relies on Python and many packages from the distro package manager. But your app could pick up one of those dependencies! At best, this will work fine. But in the worse cases, perhaps it subtly behaves differently or simply does not work at all. My understanding is that Rust will, by default, statically link all of these dependencies. This, in Python, would be like a "pex" or "par" (or one of the many other options :^)), which does make the distribution aspect much simpler. (At the cost of build-time complexity, slowdowns, and occasional incompatibility.)
  - regularfry 5y ago
- turminal 5y ago
  Cargo is a good example of a survivorship bias of sorts. It's so deeply integrated into rust that if you can't stomach it you just leave the ecosystem and everyone that remains likes it ;)
  - pdimitar 5y ago
    Hm, so what did you dislike about `cargo`?
- uranusjr 5y ago
  Rust has and “advantage” here that it’s not generally shipped with your distro’s package manager. I think my biggest problem with this article is that the distros put Python there in the first place, and all of them apply patches to make Python work how they want it. And when it doesn’t work, it’s Python’s fault…? I mean yeah Python can definitely do something to make distributing Python easier, but it can only do so much without distros’ direct involvement. I’ll add that most of Linux distro packaging contributors are generally very nice people, understand the problem at hand, and are very open to collaboration. But sometimes you see this kind of “it’s all your fault” complaints and it’s doing exactly the opposite of helping the cause.
  - drran 5y ago
    Available Packages Name : rust Version : 1.56.1 Release : 1.fc34
- mangecoeur 5y ago
  Worth noting that the PSF has no authority to tell people how to do their packaging. Rust's Cargo is 20 years newer than Python and benefits from those decades of experience. It's a very different proposition to start a new system from fresh than to try to migrate a huge and diverse community towards it.
  - regularfry 5y ago
    This is actually where a BDFL should step in. "We're doing X for Python 5, we're using it to manage the stdlib as well as community packages, everyone get ready because I have decided."
    - Macha 5y ago
      They tried this with pip. Distros ripped it out. They added the ensurepip module so users could add it after the fact. Distros ripped it out too. Arguably the advantage newer languages like Rust and Go have in this regard is they don't even consider the distro use case - you're going to get static linking and you'd better like it. Whereas Python is from an older era and tries to fit in with the local customs and so gets hammered for its inconsistency
      - regularfry 5y ago
        Yes, and that's fine. Distro python isn't for end users.
- thetwentyone 5y ago
  And Julia's Pkg! One of my primary reasons for switching away from Python to Julia was how ergonomic modern dependency management was.
roenxi 5y ago
> Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. If I can make an observation - it has nothing to do with the number of maintainers. The problem is deep, cultural and occupies a difficult space where it might be a bug or a feature. The root cause here is that the Python project, and surrounding community, have little real respect for backwards compatibility. The complexity of Python setups is driven by the need to run multiple - potentially even mutually incompatible in the case of 2.x v. 3.x - versions of Python. All languages have packaging problems, but Python is unique in my experience in the sheer number of Python installs that I need to manage simultaneously. I still have C code that works from around the time that I learned C. I'd need another Python environment installed to say the same thing about Python.
- mangecoeur 5y ago
  It really does have to do with the number of maintainers. A huge part of the work is not just building a package manager, but coordinating many stakeholders to all use it. That would also mean building a package manager that covers all of those use cases from day one - what's the point of trying to make people use something that doesn't meet their needs. These things are a huge amount of work. Unsurprisingly, other widely used languages have the same situation - dozens of build and packaging systems for C/C++, Java, Javascript...
- allen_lasn 5y ago
  Python 2 was supported for 20 years. Python 2.7 (largely compatible with other Python 2 versions - and including backported changes from Python 3.1 - was supported for 12 years). The Python project went absolutely above and beyond to support users who wanted to drag out making not particularly complex changes to their codebase for over a decade. During this same time they improved Python 3 in response to feedback and among other effects, made it less different and easier to port, from Python 2. Many common packages supported 2.7 up to its end-of-life as well. One of the reasons that Python is often a source of compatibility errors is that both distros and large standalone applications embraced Python in the early 2000s, became dependent on a particular version, and then refused to work with newer versions. Python is not responsible for all the engineering decisions everyone writing in the language has ever made.
l0b0 5y ago
I'm not saying it solves every problem for everyone, but have you looked at Nix? I've found it to be great for portability (at least on other Linux distros), and you can easily install anything from stable nixpkgs packages to Git repos. I'm currently using it to manage OS packages and development dependencies for some of the projects I work with (a couple at Work™).

microtonal 5y ago

And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems, like NixOS or OSTree-based system like Silverblue), a wide variety of build systems that sometimes hook into make, etc. Solving this is a hard problem. This is why an authority like PSF has to step in and say: this is how it is going to be done from here onwards. [1] E.g. Rust's Cargo.

komuher 5y ago
Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-
- smallerfish 5y ago
  Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.
  - RegW 5y ago
    Ant didn't do package management.
    - smallerfish 5y ago
      No, but ivy was commonly used. That said, the bigger point is that the standard way to do things can be changed. For some reason, despite (or because of?) PEP, the python community seems unable to coalesce around standard ways of packaging.
      - RegW 5y ago
        Perhaps I have lived too long out in the provences, but Maven was my first experience of dependency management in Java. After Ant it was a no-brainer because Ant didn't do dependencies. I didn't come across references to Ivy for some years and personally I have never seen it used in the wild. That said, I didn't actually like early Maven that much, it was pretty inflexible and often required the AntRun plugin to do something novel. However, it is still top dog everywhere I work and very much the standard way to do things.
- WhyNotHugo 5y ago
  Which is why I'm starting to feel like Python is the new Java.
  - oneweekwonder 5y ago
    Strange, because python was released almost 5 years before java.
    - mangecoeur 5y ago
      Python is the Old Java :P
- microtonal 5y ago
  Which is exactly what my second paragraph said :). This is a common struggle for many older programming language ecosystems, e.g. I think the same is true to some extend for C and C++. As one of the sibling commenters mentions, there are good examples that it is possible to standardize packaging better. Maven replaced IDE-driven builds and Ant for in the Java ecosystem and added proper package management. Additionally, it required that projects start conforming to standardized layouts, by taking convention over configuration and being largely declarative. I think the Maven success story lost some of its shine with Gradle, but that's another story.
- ajsnigrutin 5y ago
  Perl is 33 years old, and perl 5.x is 27 years old. I can literally take a 20+yo book and all the examples still work. CPAN still works. I literally have 20+yo scripts still copied from server to server, from laptop to laptop, without any changes.
__float 5y ago
Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version). Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, unsupported, but up to date".
- wildbook 5y ago
  Rust almost does the exact same thing, so I'd say it's fair. Dependencies (crates) are grabbed in source format and compiled locally as part of the build process, and installing Rust programs through Cargo also compiles them (and their dependencies) locally. Some crates have the same issue where build scripts rely on outside tooling being installed, but it's definitely not common to (unless you're relying on compiling C/C++ code for FFI for example, in which case it's somewhat frequent).
  - __float 5y ago
    I think there needs to be a distinction between fetching crates as source at _build time_ and what happens with Python. Python's "build time" will still require the source to be present on the target machine it's deployed to -- unfortunately these tools are complex because "packaging" is only half of the issue, it's also the distribution and _deployment_ which makes things messy. Consider that, even if you want to use packages only from your application's virtualenv, the default (footgun warning!) is that Python will still use the "system" packages -- this means you may have installed Ansible or some other tool that relies on Python and many packages from the distro package manager. But your app could pick up one of those dependencies! At best, this will work fine. But in the worse cases, perhaps it subtly behaves differently or simply does not work at all. My understanding is that Rust will, by default, statically link all of these dependencies. This, in Python, would be like a "pex" or "par" (or one of the many other options :^)), which does make the distribution aspect much simpler. (At the cost of build-time complexity, slowdowns, and occasional incompatibility.)
- regularfry 5y ago
turminal 5y ago
Cargo is a good example of a survivorship bias of sorts. It's so deeply integrated into rust that if you can't stomach it you just leave the ecosystem and everyone that remains likes it ;)
- pdimitar 5y ago
  Hm, so what did you dislike about `cargo`?
uranusjr 5y ago
Rust has and “advantage” here that it’s not generally shipped with your distro’s package manager. I think my biggest problem with this article is that the distros put Python there in the first place, and all of them apply patches to make Python work how they want it. And when it doesn’t work, it’s Python’s fault…? I mean yeah Python can definitely do something to make distributing Python easier, but it can only do so much without distros’ direct involvement. I’ll add that most of Linux distro packaging contributors are generally very nice people, understand the problem at hand, and are very open to collaboration. But sometimes you see this kind of “it’s all your fault” complaints and it’s doing exactly the opposite of helping the cause.
- drran 5y ago
  Available Packages Name : rust Version : 1.56.1 Release : 1.fc34
mangecoeur 5y ago
Worth noting that the PSF has no authority to tell people how to do their packaging. Rust's Cargo is 20 years newer than Python and benefits from those decades of experience. It's a very different proposition to start a new system from fresh than to try to migrate a huge and diverse community towards it.
- regularfry 5y ago
  This is actually where a BDFL should step in. "We're doing X for Python 5, we're using it to manage the stdlib as well as community packages, everyone get ready because I have decided."
  - Macha 5y ago
    They tried this with pip. Distros ripped it out. They added the ensurepip module so users could add it after the fact. Distros ripped it out too. Arguably the advantage newer languages like Rust and Go have in this regard is they don't even consider the distro use case - you're going to get static linking and you'd better like it. Whereas Python is from an older era and tries to fit in with the local customs and so gets hammered for its inconsistency
    - regularfry 5y ago
      Yes, and that's fine. Distro python isn't for end users.
thetwentyone 5y ago
And Julia's Pkg! One of my primary reasons for switching away from Python to Julia was how ergonomic modern dependency management was.

roenxi 5y ago

> Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. If I can make an observation - it has nothing to do with the number of maintainers. The problem is deep, cultural and occupies a difficult space where it might be a bug or a feature. The root cause here is that the Python project, and surrounding community, have little real respect for backwards compatibility. The complexity of Python setups is driven by the need to run multiple - potentially even mutually incompatible in the case of 2.x v. 3.x - versions of Python. All languages have packaging problems, but Python is unique in my experience in the sheer number of Python installs that I need to manage simultaneously. I still have C code that works from around the time that I learned C. I'd need another Python environment installed to say the same thing about Python.

mangecoeur 5y ago
It really does have to do with the number of maintainers. A huge part of the work is not just building a package manager, but coordinating many stakeholders to all use it. That would also mean building a package manager that covers all of those use cases from day one - what's the point of trying to make people use something that doesn't meet their needs. These things are a huge amount of work. Unsurprisingly, other widely used languages have the same situation - dozens of build and packaging systems for C/C++, Java, Javascript...
allen_lasn 5y ago
Python 2 was supported for 20 years. Python 2.7 (largely compatible with other Python 2 versions - and including backported changes from Python 3.1 - was supported for 12 years). The Python project went absolutely above and beyond to support users who wanted to drag out making not particularly complex changes to their codebase for over a decade. During this same time they improved Python 3 in response to feedback and among other effects, made it less different and easier to port, from Python 2. Many common packages supported 2.7 up to its end-of-life as well. One of the reasons that Python is often a source of compatibility errors is that both distros and large standalone applications embraced Python in the early 2000s, became dependent on a particular version, and then refused to work with newer versions. Python is not responsible for all the engineering decisions everyone writing in the language has ever made.

l0b0 5y ago

I'm not saying it solves every problem for everyone, but have you looked at Nix? I've found it to be great for portability (at least on other Linux distros), and you can easily install anything from stable nixpkgs packages to Git repos. I'm currently using it to manage OS packages and development dependencies for some of the projects I work with (a couple at Work™).