11 ms·

Thoughts on the Python packaging ecosystem

130 points by BerislavLopac 4y ago

DougMerritt 4y ago
At the top is a "tl;dr" which makes some points that I think are good to keep in mind for any set of tools: Pick from N different tools that do N different things is a good model. Pick from N ~equivalent choices is a really bad user experience. Picking a default doesn’t make other approaches illegal.
- zdragnar 4y ago
  As someone who has had to take over the projects of other people, I will say the third point needs a caveat: Picking a non-default tool needs a written explanation with sound reasoning When there is a default, new members to the community will gravitate towards it and, if there isn't a good reason, will avoid learning other options. In time, this ends up being anywhere from a minor annoyance to very frustrating, as not only are you losing time to learning, but potentially losing time to not having access to extras as the community around the default expands and your now special snowflake project goes without. For very specific tools and libraries, it might be okay if they are easily interchangeable (but is apparently bad UX?) But for things with larger responsibilities - a full blown ORM, package manager, etc- it can end up being a millstone.
di 4y ago
Previous discussion: https://news.ycombinator.com/item?id=34464657 https://news.ycombinator.com/item?id=34464657
chaxor 4y ago
Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The other problem is due to python's lead in ML and the need to work with GPUs. That started with tensorflow, which had immense (still does) issues in dealing with versioning, mostly due to Nvidia (matching cuda versions and such). So effectively, Nvidia and package management killed a good language.
- kelsolaar 4y ago
  Poetry certainly made our life much easier and was a productivity improvement. It the the tool that brought us back from Conda because it was able to solve environments than nothing in Pip land was able to do.
  - jvolkman 4y ago
    Was that before or after pip's new resolver was released at the end of 2020?
    - kelsolaar 4y ago
      Before!
  - totalhack 4y ago
    Poetry seems pretty slow due to some of the extra dependency logic it builds on top of pip. Whether this trade off is worth it probably depends on the scope of the project. For most of my projects I stick with pip and almost never run into issues.
  - anuraaga 4y ago
    It's unlikely Python would still be a popular language without Conda. As the article says, many Python users are not traditional software engineers - they need a tool that just works and Conda is the closest thing in the ecosystem. Used to spend much time to optimize Gentoo Linux but eventually switched to Ubuntu to be productive. I feel Conda is similarly useful, especially if not considering Docker i.e. local dev, and am disappointed hackers seem to be dismissing it.
  - chaxor 4y ago
    The problem is that poetry/conda/etc chose to make their own thing to begin with, when they should've just improved the standard language tool with the features they wanted. That's one key difference between shit languages (such as R) and languages that have good tooling (e.g. Rust) - the community works on the core together more rather than each person doing their own thing and confusing everyone in the community by not having a standard way to do anything. Every package management developer for python should have just added features to the standard package manager instead of doing their own thing.
woodruffw 4y ago
This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX.
- saghm 4y ago
  > Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX. Not to mention that all efforts would be focused on improving a single package manager. Even if the one that was standardized wasn't the "best" at first (by whichever metric you prefer), giving everyone an incentive to improve that one will likely make it the best within a few years.
- divbzero 4y ago
  To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm/yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses (e.g. a lock file) and has remained the default choice in that ecosystem. I suspect the same would happen with pip if it too made significant improvements.
  - lmm 4y ago
    Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.
atomicnumber3 4y ago
Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spoiled coming from Java land, where aside from too much XML, maven just works and if you need to do weird shit (like if you're android, for instance) then you use gradle, which still doesn't fuck up the maven repository format and we can all just live happily. (And nobody uses Ant anymore).
- Kwpolska 4y ago
  Python packaging has too many people inventing their own tools and their own little fiefdoms instead of picking one good tool and pushing it forward. There’s also the PyPA, which is an organization (or a group of semi-related volunteers, depending which interpretation suits them best) that maintains many of the tools (although none of the most feature-complete, namely poetry and PDM) and that produces standards that promote tooling proliferation.
  - wiredfool 4y ago
    I’ve just looked through the docs for poetry and I don’t see anything there about compiled extensions.
    - Kwpolska 4y ago
      The compiled extension story is pretty bad in most tools. Poetry can do it with a custom build.py file, which calls distutils/setuptools: https://stackoverflow.com/questions/60073711/how-to-build-c-extensions-via-poetry https://stackoverflow.com/questions/60073711/how-to-build-c-...
- woodruffw 4y ago
  In some ways, it's actually pretty weird that Ruby hasn't had similar problems (I say wearing both my Ruby fanboy and Python packaging contributor hats). In others, it's less weird: Ruby's upswing happened with Rails and a handful of other "killer" frameworks, which helped to solidify developer workflows (and expectations) around packaging. Python, by contrast, has had multiple generations of "killer" usecases, each with their own baggage (and many predating any real packaging standards). The history of Python packaging is also much older, and much more devolved than anybody in 2023 would consider reasonable for a packaging ecosystem: the earliest generation of PyPI, for example, was just an index that pointed to other webhosts for downloads, rather than a full package host. This helped ossify manual workflows that developers at the time were content with, and some of that cost is still being paid forwards.
ram_rar 4y ago
I get a lot of flak for saying this, but I do hope python committers and psf members take a hard look at perl and its demise. Perl had a very similar problem of having N equivalent choices for doing the same thing and python for better or worse is heading in the same direction. In the last few yrs of my professional career many projects I worked, were on are migrating python -> golang cuz frankly many people were fed up of python2->3 migration. Not that golang doesnt have it fair share of issues, but the the packaging and deploying aspects of a single binary makes CI/CD and operations a breeze. I hope python makes this is first class citizen in its ecosystem.
- KyeRussell 4y ago
  These days I don’t think I’m being too brash in saying that anyone dealing with Python 2 to Python 3 transition stuff is so deep in the realm of enterprise abandonware that it’s not worth paying it much attention. Not necessarily including you here! But as someone that went through the transition, I’m certainly not going to deny how much of a shit show it was. And I get a feeling that the two situations were/are caused by the same political / organisational / philosophical factors. I’d be miffed if the Python / PyPA mob got so distracted with their internal politics, certain terse personalities, or the hands-off competition-is-good ideology, that the packaging story makes Python an increasingly unappealing choice. Even worse, we could end up with a packaging ecosystem run by effing Microsoft like the JS people do, because “competition”. And yes I’m totally conflating aspects of packaging here. At least we all settled on PyPi, except for all those that haven’t… :)
  - at_a_remove 4y ago
    It is, sadly, not always Enterprise abandonware. The industry I am in is extremely conservative about its upgrades and changes. Between that, our custom software interfacing with the 800 pound gorilla for our market, and a few other things, we're behind. Now, this software depends on a very specific version of ArcGIS. Desktop, not Pro. Not just version 10. Not just version 10.2. But version 10.2.1. If you go look at ESRI's page for Python and 10.2.1, it says, in large and bold letters (which is something ESRI doesn't do in its documentation very often), NOT TO UPGRADE OR CHANGE PYTHON VERSIONS, EVEN A TINY BIT. And that version is 2.7. I couldn't even get pip working when I wanted to install a very old version of lxml I wanted. I guess what I am getting at is, as a language becomes successful and has market penetration, it also seeps down, way far down, in the chain of dependencies. It's the price of success, essentially. I think language maintainers should really pay more attention to that. Just as you know how a program ends up sticking around longer than you originally thought, so too do versions of your language. It is in some ways akin to trying to explain to da Vinci that very far in the future, some of his artworks will adorn clothing and coffee mugs: the good stuff just gets to places you would never dream.
philsnow 4y ago
> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are half a dozen different things for the "build" / transpile / compile stage of frontend work, and > Pick from N ~equivalent choices is a really bad user experience this has caused me to bounce off of getting into frontend work several times. It's so aesthetically displeasing that my brain doesn't want to learn it.
- jessaustin 4y ago
  ...aren't there at least two or three package managers (npm, yarn, maybe pnpm)? I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.
  - KyeRussell 4y ago
    Yes, but one has to come to that conclusion oneself, which increases friction. I realise you’re talking about pnpm, and not yarn, but when I think of issues like this, yarn is what comes to mind. A bunch of projects prefer yarn over npm, or at the very least place them side by side, in the project documentation. As someone reasonably familiar with JavaScript, but that only works with it sporadically, I always find myself going back and trying to reverse-engineer the reasoning, and I’m never entirely confident. So when I circle back to my 6-monthly journey into writing some JavaScript, and see another thing called “pnpm” making its way into package installation docs, I cannot be blamed for sighing and being a little irked. Especially since my daily driver is Python, so I’ve got loads of packaging trauma.
    - Aeolun 4y ago
      I think the nice thing about the node ecosystem is that all the packaging tools are using the exact same package.json. The only thing that’s different is the way they resolve/install the packages.
wheelerof4te 4y ago
There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip. It is de facto standard tool for installing packages. It can be extended to do more. Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer systems programming languages. We need to learn from Java's and Javascript's ecosystems. Those languages are now used mostly in their pure form. They are portable, optimized and more stable than most languages. With enough care, Python can become the same.
- Kamq 4y ago
  > Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc. > Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer systems programming languages. I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade. I mean, this is once of python's main selling points. It doesn't matter what your numeric/scientific library is written in, python can call it, and it's vaguely convenient to write (depending on your taste for comprehensions). Why would the python ecosystem shoot themselves in the foot like that?
  - wheelerof4te 4y ago
    "Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc." What's Django, then? "I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade." Python can only call C code. The fact that Rust itself (or Fortran, btw) has good C ABI is on Rust, not Python. Also, by calling Rust's C ABI, you are making a double indirection, thus severely reducing performance. That's why I said we need to optimize Python, and even make progress towards making Python - in Python itself. Just like Go can compile itself, Python should be able to run itself. But, that is all for the long run.
revskill 4y ago
As i see, most of issues is related to native extensions ? So it belongs to another issue instead of just packaging tool ?
- wheelerof4te 4y ago
  Indeed. Packaging pure Python often boils down to where to place the packages, the hierarchy between them and where to find the starting point for an application (if it is one and not a library). Insert native extensions into the mix and you get all kinds of issues. Like, which library to link to? Which version? Which foreign function library to use? Will my Python version be compatible with that library? Et cetera.
- ivoflipse 4y ago
  Here's an entire website dedicated to these challenges: https://pypackaging-native.github.io/ https://pypackaging-native.github.io/ Written by one of the maintainers of scipy
solarkraft 4y ago
I don't know, I'm pretty happy with Pipenv. It's the standard among the people I talk to and very good at being "pip, but better". I've tried poetry but don't really find it good enough to justify being so different, maybe unless you already have a pyproject.toml anyway. There's a lot of talk about Python packaging but as a semi-casual user I've actually had very few problems with it.
- PrivateButts 4y ago
  Pipenv left a bitter taste in many mouths, but honestly for the last two years it's been fine. What I want is for the Pipfile to get merged into the pyproject.toml at this point, so that we can speed up the move to pyproject.toml for everything.
- Hackbraten 4y ago
  Former heavy Pipenv user here. Moved all my projects to Poetry for reasons. But in hindsight, I must say that Pipenv used to work just fine for me, as does Poetry today. No major issues so far with either tool. What I do miss is being able to `pipenv run myscript`. That UX has degraded slightly for me, having to say `poetry run poe myscript` now. Barely an inconvenience though.
rerdavies 4y ago
Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?
- jopython 4y ago
  Second that. Pip has been the default for more than a decade. And yes it does have issues. Why not fix it or add options to pip instead of piling on more tools?
  - bandrami 4y ago
    Because for whatever reason Python really attracts the Start Over And This Time We’ll Get It Right crowd. It’s why I don’t start new projects with it anymore, but make most of my living cleaning up the mess that incompatible changes make with existing software. So, in that sense, it’s great for keeping devops employed.
- ehutch79 4y ago
  There’s nothing really wrong with pip. It doesn’t have a lot of features, but its position isn’t one to have those features. It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going
  - rerdavies 4y ago
    Keras/tensorflow has some <= version specifications for modules it depends on. If there is a > version installed, pip install fails. Being able to install multiple versions of the same module seems like a fairly basic feature for a packaging manager. And it would seem to me that having a shim executable to examine a profile of some kind for an app at startup, and load/install the correct version of python doesn't seem impossible.
    - ehutch79 4y ago
      I find having multiple versions weird. That’s gotta be a niche need. If nothing else it seems like a good way to let packages be lax in keeping their dependencies up to date.
pronlover723 4y ago
For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get out of the default "globals" and you need other incantations to switch projects.
- ivoflipse 4y ago
  Lucky for you, there's a new PEP proposing just that: https://discuss.python.org/t/pep-704-require-virtual-environments-by-default-for-package-installers/22846/61 https://discuss.python.org/t/pep-704-require-virtual-environ... Written by the author of this blog post no less! However, you'll find that as with all packaging discussions there are people opposing it, because their workflow doesn't match yours and they don't want to change how they work. We, as a community, need a way to resolve such stalemates or I fear we won't make much headway
  - earthboundkid 4y ago
    We need some person, a nice person, a benevolent person, who could some how tell other people what to do, dictate it if you will, and it would be best if they could keep up this job for the rest of his or her life. We’ll call them, the Friendly Language Uncle.
    - numbsafari 4y ago
      I’d prefer a crazy aunt to a creepy uncle… but maybe let’s leave family out of it altogether.
      - uranusjr 4y ago
        Parent is referring to a long-running (but somewhat obscure at this point unfortunately) joke https://peps.python.org/pep-0401/ https://peps.python.org/pep-0401/
fumeux_fume 4y ago
Python has its warts and this one of them, but I think people embellish how bad it is. Perhaps the reason why there isn't a singular packaging solution like other languages is because most Python users get along just fine with the alternatives that exist.
cdogl 4y ago
The (absent) packaging system is the single greatest reason I’ve moved away from python professionally. Life is too short to do without vendoring and/or lock files - especially in an interpreted language.
woolion 4y ago
Very good summary of the situation, I hope something good will come out of it. I've used pipenv, pyenv, poetry and settled for my own use-cases on just pip with virtualenvwrapper. So any program goes into its dedicated folder with requirements and virtualenv. However there's something that I've never managed to get working. You have say a 3.8 virtualenv, and package file mentions python should be >=3.8. But when calling python -m build, the build process creates a new environment using the system python. So if you are on an old system that is on Python 3.7, the resulting package cannot be installed because of the inconsistency between declared python version and the one that has been used. It may be a very stupid problem, but I haven't been able to find any documentation for it. Any search engine is just too happy to throw any 'python packaging result' before anything so specific. Maybe I should use one other tool for that? But there's no clear (default?) path documented.
- ivoflipse 4y ago
  You could look into cibuildwheel, to have more control over what Python versions are used to build a wheel, but as the name implies not a great solution for running it locally. Otherwise you'll have to be rather specific about what Python interpreter to use. For example using the python launcher for Linux (https://github.com/brettcannon/python-launcher https://github.com/brettcannon/python-launcher). Looking through the docs of pdm, Hatch or Poetry, I can't really find a definitive answer if they will use the Python version you specify. They all at least need to be able to locate the correct Python version in order to do so. It would be great if managing Python versions was something that came along with Python, so that these tools could rely on it more.
Too 4y ago
I'm convinced 99% of all issues with pip can be solved with 3 simple aliases. It already has everything else, it just the UX that isn't opinionated enough. There wouldn't be a need for Pipenv, Poetry or yet another project-definition or lockfile format, if only these few fundamental commands with sane defaults were included out of the box. 1. Installation. Just install everything in a bloody application-specific virtualenv by default and use the existing lockfile mechanism by default. pyinstall='python -m venv .pip_modules && ./.pip_modules/bin/activate && pip install --editable ./ --constraints=constraints.txt' 2. Lockfiles. Encourage and always always use the constraints.txt-file. Maybe take the oppurtunity to rename it to lock.pip instead. pyfreeze='./pip_modules/bin/pip freeze > lock.pip' 3. Running. Remembering to activate a virtualenv is too easy to forget and leads to mode-confusion. We need an equivalent of npm run. pyrun='./pip_modules/bin/$1' Done. That took 5 minutes tops. Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years. It seems they are more busy bikeshedding forever about the build-backends or whatever else that only very few packagers cares about. The packaging of course has its issues but they are all minor compared to preserving end-users sanity. The UX is the #1 issue that impacts millions of users daily, it can and just need to be fixed yesterday.
- Hackbraten 4y ago
  > Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years. My feeling is that that approach would cause dozens of itches to pop up. Things like, “my coworker is on Windows and I’m struggling to come up with step-by-step instructions for her to set up the project.” Those are valid UX issues but likely just the beginning of a rabbit hole. I think that tackling those issues would inevitably lead to yet another Poetry/Pipenv clone – because with UX, the devil is in the details.
egberts1 4y ago
I think I can safely speak for most everyone: I await for that one build tool to rule them all things Python; and an HOWTO that covers them all.

130 points by BerislavLopac 4y ago

At the top is a "tl;dr" which makes some points that I think are good to keep in mind for any set of tools: Pick from N different tools that do N different things is a good model. Pick from N ~equivalent choices is a really bad user experience. Picking a default doesn’t make other approaches illegal.

zdragnar 4y ago
As someone who has had to take over the projects of other people, I will say the third point needs a caveat: Picking a non-default tool needs a written explanation with sound reasoning When there is a default, new members to the community will gravitate towards it and, if there isn't a good reason, will avoid learning other options. In time, this ends up being anywhere from a minor annoyance to very frustrating, as not only are you losing time to learning, but potentially losing time to not having access to extras as the community around the default expands and your now special snowflake project goes without. For very specific tools and libraries, it might be okay if they are easily interchangeable (but is apparently bad UX?) But for things with larger responsibilities - a full blown ORM, package manager, etc- it can end up being a millstone.

di 4y ago

Previous discussion: https://news.ycombinator.com/item?id=34464657 https://news.ycombinator.com/item?id=34464657

chaxor 4y ago

Packaging and Nvidia are pretty much the main reasons python has degraded in status for the past 7 years. It started with the added and unnecessary confusion from conda (and silly things like pythonxy), but has really escalated through extra stupidities like poetry. Much of the features that may be enticing for these extra packages should have pushed the developers to improve the standard tools in python (pip). The other problem is due to python's lead in ML and the need to work with GPUs. That started with tensorflow, which had immense (still does) issues in dealing with versioning, mostly due to Nvidia (matching cuda versions and such). So effectively, Nvidia and package management killed a good language.

kelsolaar 4y ago
Poetry certainly made our life much easier and was a productivity improvement. It the the tool that brought us back from Conda because it was able to solve environments than nothing in Pip land was able to do.
- jvolkman 4y ago
  Was that before or after pip's new resolver was released at the end of 2020?
  - kelsolaar 4y ago
    Before!
- totalhack 4y ago
  Poetry seems pretty slow due to some of the extra dependency logic it builds on top of pip. Whether this trade off is worth it probably depends on the scope of the project. For most of my projects I stick with pip and almost never run into issues.
- anuraaga 4y ago
  It's unlikely Python would still be a popular language without Conda. As the article says, many Python users are not traditional software engineers - they need a tool that just works and Conda is the closest thing in the ecosystem. Used to spend much time to optimize Gentoo Linux but eventually switched to Ubuntu to be productive. I feel Conda is similarly useful, especially if not considering Docker i.e. local dev, and am disappointed hackers seem to be dismissing it.
- chaxor 4y ago
  The problem is that poetry/conda/etc chose to make their own thing to begin with, when they should've just improved the standard language tool with the features they wanted. That's one key difference between shit languages (such as R) and languages that have good tooling (e.g. Rust) - the community works on the core together more rather than each person doing their own thing and confusing everyone in the community by not having a standard way to do anything. Every package management developer for python should have just added features to the standard package manager instead of doing their own thing.

woodruffw 4y ago

This is a great writeup by a central figure in Python packaging, and gets to the core of one of Python packaging's biggest strengths (and weaknesses): the PyPA is primarily an "open tent," with mostly independent (but somewhat standards-driven) development within it. Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX.

saghm 4y ago
> Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX. Not to mention that all efforts would be focused on improving a single package manager. Even if the one that was standardized wasn't the "best" at first (by whichever metric you prefer), giving everyone an incentive to improve that one will likely make it the best within a few years.
divbzero 4y ago
To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today. The npm/yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses (e.g. a lock file) and has remained the default choice in that ecosystem. I suspect the same would happen with pip if it too made significant improvements.
- lmm 4y ago
  Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

atomicnumber3 4y ago

Why does Ruby not have similar problems? (Or if it does, why does nobody seem to care?) Like why does Ruby have gem and bundler, each doing one thing, whereas python has fifty bazillion tools that all do nearly the same thing if you squint but all have their own weird problems? I've personally just ended up using poetry and that has mostly stopped me from having to care overmuch about the tooling. I feel like I'm spoiled coming from Java land, where aside from too much XML, maven just works and if you need to do weird shit (like if you're android, for instance) then you use gradle, which still doesn't fuck up the maven repository format and we can all just live happily. (And nobody uses Ant anymore).

Kwpolska 4y ago
Python packaging has too many people inventing their own tools and their own little fiefdoms instead of picking one good tool and pushing it forward. There’s also the PyPA, which is an organization (or a group of semi-related volunteers, depending which interpretation suits them best) that maintains many of the tools (although none of the most feature-complete, namely poetry and PDM) and that produces standards that promote tooling proliferation.
- wiredfool 4y ago
  I’ve just looked through the docs for poetry and I don’t see anything there about compiled extensions.
  - Kwpolska 4y ago
    The compiled extension story is pretty bad in most tools. Poetry can do it with a custom build.py file, which calls distutils/setuptools: https://stackoverflow.com/questions/60073711/how-to-build-c-extensions-via-poetry https://stackoverflow.com/questions/60073711/how-to-build-c-...
woodruffw 4y ago
In some ways, it's actually pretty weird that Ruby hasn't had similar problems (I say wearing both my Ruby fanboy and Python packaging contributor hats). In others, it's less weird: Ruby's upswing happened with Rails and a handful of other "killer" frameworks, which helped to solidify developer workflows (and expectations) around packaging. Python, by contrast, has had multiple generations of "killer" usecases, each with their own baggage (and many predating any real packaging standards). The history of Python packaging is also much older, and much more devolved than anybody in 2023 would consider reasonable for a packaging ecosystem: the earliest generation of PyPI, for example, was just an index that pointed to other webhosts for downloads, rather than a full package host. This helped ossify manual workflows that developers at the time were content with, and some of that cost is still being paid forwards.

ram_rar 4y ago

I get a lot of flak for saying this, but I do hope python committers and psf members take a hard look at perl and its demise. Perl had a very similar problem of having N equivalent choices for doing the same thing and python for better or worse is heading in the same direction. In the last few yrs of my professional career many projects I worked, were on are migrating python -> golang cuz frankly many people were fed up of python2->3 migration. Not that golang doesnt have it fair share of issues, but the the packaging and deploying aspects of a single binary makes CI/CD and operations a breeze. I hope python makes this is first class citizen in its ecosystem.

KyeRussell 4y ago
These days I don’t think I’m being too brash in saying that anyone dealing with Python 2 to Python 3 transition stuff is so deep in the realm of enterprise abandonware that it’s not worth paying it much attention. Not necessarily including you here! But as someone that went through the transition, I’m certainly not going to deny how much of a shit show it was. And I get a feeling that the two situations were/are caused by the same political / organisational / philosophical factors. I’d be miffed if the Python / PyPA mob got so distracted with their internal politics, certain terse personalities, or the hands-off competition-is-good ideology, that the packaging story makes Python an increasingly unappealing choice. Even worse, we could end up with a packaging ecosystem run by effing Microsoft like the JS people do, because “competition”. And yes I’m totally conflating aspects of packaging here. At least we all settled on PyPi, except for all those that haven’t… :)
- at_a_remove 4y ago
  It is, sadly, not always Enterprise abandonware. The industry I am in is extremely conservative about its upgrades and changes. Between that, our custom software interfacing with the 800 pound gorilla for our market, and a few other things, we're behind. Now, this software depends on a very specific version of ArcGIS. Desktop, not Pro. Not just version 10. Not just version 10.2. But version 10.2.1. If you go look at ESRI's page for Python and 10.2.1, it says, in large and bold letters (which is something ESRI doesn't do in its documentation very often), NOT TO UPGRADE OR CHANGE PYTHON VERSIONS, EVEN A TINY BIT. And that version is 2.7. I couldn't even get pip working when I wanted to install a very old version of lxml I wanted. I guess what I am getting at is, as a language becomes successful and has market penetration, it also seeps down, way far down, in the chain of dependencies. It's the price of success, essentially. I think language maintainers should really pay more attention to that. Just as you know how a program ends up sticking around longer than you originally thought, so too do versions of your language. It is in some ways akin to trying to explain to da Vinci that very far in the future, some of his artworks will adorn clothing and coffee mugs: the good stuff just gets to places you would never dream.

philsnow 4y ago

> A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), [...], etc) – a single tool that provides a build system, dependency manager, publishing, running project-specific tasks/scripts, etc. I’ve referred to this as “workflow tool” in this post. I don't know enough about node, but aren't there at least two or three package managers (npm, yarn, maybe pnpm)? Then there are half a dozen different things for the "build" / transpile / compile stage of frontend work, and > Pick from N ~equivalent choices is a really bad user experience this has caused me to bounce off of getting into frontend work several times. It's so aesthetically displeasing that my brain doesn't want to learn it.

jessaustin 4y ago
...aren't there at least two or three package managers (npm, yarn, maybe pnpm)? I'm not sure about yarn, but pnpm has exactly the same API as npm has. It simply has a different disk organization and caching strategy. If npm maintainers so chose, pnpm's behavior would be an option within npm. Since they haven't chosen that, it seems completely reasonable to "compete" in the way that pnpm does.
- KyeRussell 4y ago
  Yes, but one has to come to that conclusion oneself, which increases friction. I realise you’re talking about pnpm, and not yarn, but when I think of issues like this, yarn is what comes to mind. A bunch of projects prefer yarn over npm, or at the very least place them side by side, in the project documentation. As someone reasonably familiar with JavaScript, but that only works with it sporadically, I always find myself going back and trying to reverse-engineer the reasoning, and I’m never entirely confident. So when I circle back to my 6-monthly journey into writing some JavaScript, and see another thing called “pnpm” making its way into package installation docs, I cannot be blamed for sighing and being a little irked. Especially since my daily driver is Python, so I’ve got loads of packaging trauma.
  - Aeolun 4y ago
    I think the nice thing about the node ecosystem is that all the packaging tools are using the exact same package.json. The only thing that’s different is the way they resolve/install the packages.

wheelerof4te 4y ago

There really needs to be one tool that combines installing, building, testing and running Python projects. This tool should be officially endorsed by core developers of Python and gradually added to the standard distribution model. The good starting point for this would be Pip. It is de facto standard tool for installing packages. It can be extended to do more. Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer systems programming languages. We need to learn from Java's and Javascript's ecosystems. Those languages are now used mostly in their pure form. They are portable, optimized and more stable than most languages. With enough care, Python can become the same.

Kamq 4y ago
> Also, the standard should favor pure-Python packages, in order to untangle Python from it's legacy as a glue language for modules written in C(++). Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc. > Otherwise, no progress will be made at the language level and Python will die out once C becomes legacy language, replaced by safer systems programming languages. I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade. I mean, this is once of python's main selling points. It doesn't matter what your numeric/scientific library is written in, python can call it, and it's vaguely convenient to write (depending on your taste for comprehensions). Why would the python ecosystem shoot themselves in the foot like that?
- wheelerof4te 4y ago
  "Never gonna happen. Python's explosion has been through data science and ml, which is based entirely around calling Fortran/C/C++/etc." What's Django, then? "I assume you mean Rust here. Do you think python can't already call rust? It's been able to do that for the better part of a decade." Python can only call C code. The fact that Rust itself (or Fortran, btw) has good C ABI is on Rust, not Python. Also, by calling Rust's C ABI, you are making a double indirection, thus severely reducing performance. That's why I said we need to optimize Python, and even make progress towards making Python - in Python itself. Just like Go can compile itself, Python should be able to run itself. But, that is all for the long run.

revskill 4y ago

As i see, most of issues is related to native extensions ? So it belongs to another issue instead of just packaging tool ?

wheelerof4te 4y ago
Indeed. Packaging pure Python often boils down to where to place the packages, the hierarchy between them and where to find the starting point for an application (if it is one and not a library). Insert native extensions into the mix and you get all kinds of issues. Like, which library to link to? Which version? Which foreign function library to use? Will my Python version be compatible with that library? Et cetera.
ivoflipse 4y ago
Here's an entire website dedicated to these challenges: https://pypackaging-native.github.io/ https://pypackaging-native.github.io/ Written by one of the maintainers of scipy

solarkraft 4y ago

I don't know, I'm pretty happy with Pipenv. It's the standard among the people I talk to and very good at being "pip, but better". I've tried poetry but don't really find it good enough to justify being so different, maybe unless you already have a pyproject.toml anyway. There's a lot of talk about Python packaging but as a semi-casual user I've actually had very few problems with it.

PrivateButts 4y ago
Pipenv left a bitter taste in many mouths, but honestly for the last two years it's been fine. What I want is for the Pipfile to get merged into the pyproject.toml at this point, so that we can speed up the move to pyproject.toml for everything.
Hackbraten 4y ago
Former heavy Pipenv user here. Moved all my projects to Poetry for reasons. But in hindsight, I must say that Pipenv used to work just fine for me, as does Poetry today. No major issues so far with either tool. What I do miss is being able to `pipenv run myscript`. That UX has degraded slightly for me, having to say `poetry run poe myscript` now. Barely an inconvenience though.

rerdavies 4y ago

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

jopython 4y ago
Second that. Pip has been the default for more than a decade. And yes it does have issues. Why not fix it or add options to pip instead of piling on more tools?
- bandrami 4y ago
  Because for whatever reason Python really attracts the Start Over And This Time We’ll Get It Right crowd. It’s why I don’t start new projects with it anymore, but make most of my living cleaning up the mess that incompatible changes make with existing software. So, in that sense, it’s great for keeping devops employed.
ehutch79 4y ago
There’s nothing really wrong with pip. It doesn’t have a lot of features, but its position isn’t one to have those features. It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going
- rerdavies 4y ago
  Keras/tensorflow has some <= version specifications for modules it depends on. If there is a > version installed, pip install fails. Being able to install multiple versions of the same module seems like a fairly basic feature for a packaging manager. And it would seem to me that having a shim executable to examine a profile of some kind for an app at startup, and load/install the correct version of python doesn't seem impossible.
  - ehutch79 4y ago
    I find having multiple versions weird. That’s gotta be a niche need. If nothing else it seems like a good way to let packages be lax in keeping their dependencies up to date.

pronlover723 4y ago

For me, before I even get to packaging, I hit the install/environment issue with python. Python, by default, wants to be installed at a system level and wants libraries/packages to be at a system level. That shit has to stop. The default needs to be project local installs. Node might have issues but one thing they got right is defaulting to project local installs vs python where you need various incantations to get out of the default "globals" and you need other incantations to switch projects.

ivoflipse 4y ago
Lucky for you, there's a new PEP proposing just that: https://discuss.python.org/t/pep-704-require-virtual-environments-by-default-for-package-installers/22846/61 https://discuss.python.org/t/pep-704-require-virtual-environ... Written by the author of this blog post no less! However, you'll find that as with all packaging discussions there are people opposing it, because their workflow doesn't match yours and they don't want to change how they work. We, as a community, need a way to resolve such stalemates or I fear we won't make much headway
- earthboundkid 4y ago
  We need some person, a nice person, a benevolent person, who could some how tell other people what to do, dictate it if you will, and it would be best if they could keep up this job for the rest of his or her life. We’ll call them, the Friendly Language Uncle.
  - numbsafari 4y ago
    I’d prefer a crazy aunt to a creepy uncle… but maybe let’s leave family out of it altogether.
    - uranusjr 4y ago
      Parent is referring to a long-running (but somewhat obscure at this point unfortunately) joke https://peps.python.org/pep-0401/ https://peps.python.org/pep-0401/

fumeux_fume 4y ago

Python has its warts and this one of them, but I think people embellish how bad it is. Perhaps the reason why there isn't a singular packaging solution like other languages is because most Python users get along just fine with the alternatives that exist.

cdogl 4y ago

The (absent) packaging system is the single greatest reason I’ve moved away from python professionally. Life is too short to do without vendoring and/or lock files - especially in an interpreted language.

woolion 4y ago

Very good summary of the situation, I hope something good will come out of it. I've used pipenv, pyenv, poetry and settled for my own use-cases on just pip with virtualenvwrapper. So any program goes into its dedicated folder with requirements and virtualenv. However there's something that I've never managed to get working. You have say a 3.8 virtualenv, and package file mentions python should be >=3.8. But when calling python -m build, the build process creates a new environment using the system python. So if you are on an old system that is on Python 3.7, the resulting package cannot be installed because of the inconsistency between declared python version and the one that has been used. It may be a very stupid problem, but I haven't been able to find any documentation for it. Any search engine is just too happy to throw any 'python packaging result' before anything so specific. Maybe I should use one other tool for that? But there's no clear (default?) path documented.

ivoflipse 4y ago
You could look into cibuildwheel, to have more control over what Python versions are used to build a wheel, but as the name implies not a great solution for running it locally. Otherwise you'll have to be rather specific about what Python interpreter to use. For example using the python launcher for Linux (https://github.com/brettcannon/python-launcher https://github.com/brettcannon/python-launcher). Looking through the docs of pdm, Hatch or Poetry, I can't really find a definitive answer if they will use the Python version you specify. They all at least need to be able to locate the correct Python version in order to do so. It would be great if managing Python versions was something that came along with Python, so that these tools could rely on it more.

Too 4y ago

I'm convinced 99% of all issues with pip can be solved with 3 simple aliases. It already has everything else, it just the UX that isn't opinionated enough. There wouldn't be a need for Pipenv, Poetry or yet another project-definition or lockfile format, if only these few fundamental commands with sane defaults were included out of the box. 1. Installation. Just install everything in a bloody application-specific virtualenv by default and use the existing lockfile mechanism by default. pyinstall='python -m venv .pip_modules && ./.pip_modules/bin/activate && pip install --editable ./ --constraints=constraints.txt' 2. Lockfiles. Encourage and always always use the constraints.txt-file. Maybe take the oppurtunity to rename it to lock.pip instead. pyfreeze='./pip_modules/bin/pip freeze > lock.pip' 3. Running. Remembering to activate a virtualenv is too easy to forget and leads to mode-confusion. We need an equivalent of npm run. pyrun='./pip_modules/bin/$1' Done. That took 5 minutes tops. Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years. It seems they are more busy bikeshedding forever about the build-backends or whatever else that only very few packagers cares about. The packaging of course has its issues but they are all minor compared to preserving end-users sanity. The UX is the #1 issue that impacts millions of users daily, it can and just need to be fixed yesterday.

Hackbraten 4y ago
> Now if you were to do this just a little more seriously its of course a bit more effort but still within the realms of weeks/months, not years. My feeling is that that approach would cause dozens of itches to pop up. Things like, “my coworker is on Windows and I’m struggling to come up with step-by-step instructions for her to set up the project.” Those are valid UX issues but likely just the beginning of a rabbit hole. I think that tackling those issues would inevitably lead to yet another Poetry/Pipenv clone – because with UX, the devil is in the details.

egberts1 4y ago

I think I can safely speak for most everyone: I await for that one build tool to rule them all things Python; and an HOWTO that covers them all.