9 ms·

Python Has Too Many Package Managers

131 points by LarsDu88 2y ago

LarsDu88 2y ago
Python actually now has over a dozen package managers, none of which are as good as Cargo in Rust. Here's my attempt at a nearly comprehensive rundown.
- eesmith 2y ago
  "like PyPy" -> "like PyPI".
  - LarsDu88 2y ago
    fixed
foundart 2y ago
Package management has been a problem almost every time I have dabbled in Python. This is a great overview of the situation and will save me time the next time.
- big-green-man 2y ago
  It didn't used to be like this. I remember first starting in Python, around a decade ago, when Python3 has been out a long time but everyone still wrote in Python2 for some reason. You used pip, that's it. Everything was easy. There were very good reasons to make some of the changes that have been made, but I think that big switch kind of normalized breaking changes in the python world. At this point we have a catastrophe on our hands. A language and it's tooling is supposed to get out of my way, it's supposed to be the tool by which I express my intention, not be a thing I have to tinker with all day. There's room for being opinionated, and there's room for upgrading things, but if I need to follow 10 RSS feeds just to keep up to date with changes, if I'm arguing with my colleagues about which of a dozen ways to use a language is best, something has gone horribly wrong.
odie5533 2y ago
I disagree with the title, but it's an okay rundown of the various package managers. Wish the author had tried out hatch though since it seems good. Also rye is not abandoned. You can see the repo has updates within the last 24 hours for a new release. I think they want uv to eat rye, but that hasn't happened yet. My current favorites are uv + mise. Handles lockfiles, multiple versions of python, and it's very fast since uv is very fast. Have not tried pdm or hatch though.
- PaulHoule 2y ago
  uv is transformative because it is (1) correct and (2) crazy fast. I wanted to like Poetry but the performance is atrocious. On the other hand, with uv, I feel like I can always build a new instance of a system whenever I want, whereas with conda or poetry I might have to wait ten minutes.
  - heyoni 2y ago
    I think the company that makes UV hired the guy that created ripgrep so no wonder it’s so fast (burntsushi i think is his alias)
- deleted 2y ago
  [deleted]
- alanwreath 2y ago
  you mean it is in the process of happening? From this article it looks like Armin R. and Charlie M. are updating Rye use uv under the hood. https://astral.sh/blog/uv https://astral.sh/blog/uv > As part of this release, we're also taking stewardship of Rye, an experimental Python packaging tool from Armin Ronacher. We'll maintain Rye as we expand uv into a unified successor project, to fulfill our shared vision for Python packaging.
gjvc 2y ago
usual bullshite about pip and venv by people who think the "activate" script does something useful
igorguerrero 2y ago
Someone from Rust talking about these tools being slow... That's rich.
- deleted 2y ago
  [deleted]
actionfromafar 2y ago
After all, the motto is "there's more than one way to do it"!
- rilindo 2y ago
  That is actually Perl's motto. [0] [0] https://wiki.c2.com/?ThereIsMoreThanOneWayToDoIt https://wiki.c2.com/?ThereIsMoreThanOneWayToDoIt
  - binary132 2y ago
    I think that’s the joke
  - drewcoo 2y ago
    But Perl started this whole one package manager to bind us thing with CPAN.
- theideaofcoffee 2y ago
  Maybe it's just a joke, but for those that don't know that is Perl's motto and even better because there's the venerable, wonderful CPAN [0] which is the de facto package repository for the language. [0] https://www.cpan.org/ https://www.cpan.org/
  - deleted 2y ago
    [deleted]
  - marcosdumay 2y ago
    It's certainly a joke, because the Python original motto was "there's only one way to do it", and the current one is "there's only one (obvious) way to do it". The Python's motto was created as an obvious reference to the Perl's one, purposefully negating it.
    - theideaofcoffee 2y ago
      Thank you for that context!
AtlasBarfed 2y ago
"Finally, because dependency resolution is a directed acylic graph (DAG)" ... Is it? :-) Mwahahaha
- glitchc 2y ago
  They forgot the word "ideally"
- stairlane 2y ago
  We analyze firmware and binaries, amongst other things at my day job. Reconstructing dependency graphs is a necessary part of our day to day. Assuming dependency graphs cant have cycles has shot us in the foot. There are absolutely real circular dependencies in assets in the wild.
  - LarsDu88 2y ago
    Original author here... wait, what? I believe pip (or at least older versions of pip) makes it possible to have cycles. Are you analyzing dep graphs made with older versions of pip?
    - stairlane 2y ago
      Not pip, but we ran into an older version of glibc that had real circular deps. Sorry I don’t remember off the top of my head which one.
- rzzzt 2y ago
  "Hey Vsauce, Michael here." https://youtu.be/TN25ghkfgQA https://youtu.be/TN25ghkfgQA
sealedservant 2y ago
I concur and I also think that there are too many build backends. pdm is my current favorite package manager. It is fully PEP-compliant and the lockfile generation is nice. I wouldn't call hatch a package manager because I don't think it can make lockfiles. uv is on my radar but it doesn't look ready for primetime yet. I saw they are building out a package management API with commands such as `uv add` and `uv remove`. Cross-platform lockfiles, editable installs, in-project .venv, and a baked-in build backend might be enough for me to make the switch. It's my pipe dream to get the full build/install/publish workflow down to a single static binary with no dependencies. Anna-Lee Popkes has an awesome comparison of the available package managers [0] complete with handy Venn diagrams. The pyOpenSci team has another nice comparison of the available build backends [1]. [0] https://alpopkes.com/posts/python/packaging_tools/ https://alpopkes.com/posts/python/packaging_tools/ [1] https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html https://www.pyopensci.org/python-package-guide/package-struc...
- PrivateButts 2y ago
  pdm is my go to as well. I've decided to really jump on the pyproject.toml train and pdm plays very well with it. .venv by default is pretty nice as well. Anything, and I mean anything, is better than pipenv though.
- mih 2y ago
  Another user of pdm here for professional projects. It sure is more standards compliant than poetry. Support for in-project venvs and integration of configs for packages such as pytest is quite useful. When evaluating package managers, poetry for sure was a contender. However listening to others experiences regarding poetry developers introducing breaking changes that could potentially cause the CI pipeline made it a no go [1]. uv seems to be coming along rather nicely, but wasn't anywhere near the level of stability compared to pdm during the evaluation phase. [1] https://www.youtube.com/watch?v=Gr9o8MW_pb0 https://www.youtube.com/watch?v=Gr9o8MW_pb0
  - odie5533 2y ago
    Poetry supports in-project venvs. You can enable it globally. https://python-poetry.org/docs/configuration/#virtualenvsin-project https://python-poetry.org/docs/configuration/#virtualenvsin-...
p1esk 2y ago
Looks like Conda is still the best package/env manager for ML engineers.
- dmd 2y ago
  Unfortunately I'm still waiting for "Solving environment". It's been 8 years. Please send help. I'm so hungry. (Yes, I know about mamba.)
  - LarsDu88 2y ago
    I lol'd at this. At my last job, we lost days if not weeks to "solving environment" due to poetry and many folks not understanding that ceiling pinning is BAD when you use a tool like poetry to manage your deps.
- hrpnk 2y ago
  Poetry user here. What is so special about Conda that makes it particularly good for ML use cases?
  - p1esk 2y ago
    No idea, I just summarized TFA.
- gillesjacobs 2y ago
  ML engineer, I am up to 3 irreversibly broken environments after simply adding a package. Conda is inexcusably slow and but even mamba can save it. Every conda project I have ever was later switched to poetry. Here's a fun challenge: try to determine channel priority in your conda env, go ahead try. Conda is the single worst packaging tool I have ever used.
  - p1esk 2y ago
    OK, would you say poetry is the best for ML engineers?
wantsanagent 2y ago
I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you that thirty years of tooling doesn't have a solution to the problem you're facing and you need this new shiny thing.* *Playing with shiny things is fun and should not be discouraged, but must not be mandated either.
- colemannerd 2y ago
  BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.
  - yjftsjthsd-h 2y ago
    > For me, this setup has caused production outages multiple times across multiple teams How does that happen?
    - colemannerd 2y ago
      I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.
      - yjftsjthsd-h 2y ago
        The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?
slt2021 2y ago
I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need. This is similar approach to “zero-based budgeting”. It forces you to carefully pick your dependencies and think about what you carry. I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade If you dont do that, you can quickly go down the javascript’s path of bloated node_modules. Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? Even venv is not really required if you dockerize your python apps, which you will have to do anyways at deploy time
- BerislavLopac 2y ago
  > Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? It's very simple: 1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.
- krapht 2y ago
  Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems. Conda solves this pain point by being a general purpose package manager. Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.
- sealedservant 2y ago
  Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.
SethMLarson 2y ago
Happily using pip, venv, and pip-tools for every project and still finding them more than suitable. They might not have the marketing budget or pizazz of others, but if you're looking for effective and boring tools that get the job done so you can solve more interesting problems they work just fine.
- verst 2y ago
  Same. Additionally I use pyenv and pyenv-virtualenv to manage multiple Python versions. Different venvs pointing to different Python versions etc, but the core tools are still pip, venv and pip-tools. (I do not use conda) I publish a Python SDK with about 85k monthly downloads according to PyPI. I make sure to run tests (unit tests, type checking, integration tests) against all currently supported minor versions of Python (3.8 - 3.12)
  - epistasis 2y ago
    I think pyenv finally solved the versioning problem for me. Additionally, being able to set a Python version or virtual environment per-directory with "pyenv local" which has eliminated having to remember which venv I was using on a project directory, or remembering which convention for venvs I used for a project.
    - verst 2y ago
      And "pyenv shell" is also handy for temporarily changing the default Python version of your current shell session. Or changing the global default using "pyenv global" - for example I currently have this set to "3.12.3", even though I usually test the SDKs I build against the oldest version I have to support first - the latest 3.8.X
VagabundoP 2y ago
Use Rye. It wasn't abandoned it ownership was transferred. Rye uses other pretty standard stuff under the hood, tools that follow PEPs, its just a front end that is sane. uv is fast as well. It downloads the pinned version of standalone Python, it keeps everything in its own venv and theres very little messing/tweaking of the environment. It is messy, although its getting better. I doubt everything will ever standardise to one tool however.
- LarsDu88 2y ago
  I made a mistake about Rye when I posted the first draft of this article. That has since been corrected. Rye is incredibly young, and when I first encountered it, I looked at its source code and found it was simply calling piptools under the hood. I did not realize that it had changed and now uses uv (or perhaps just the same solver as uv) under the hood.
  - VagabundoP 2y ago
    It is young, and it does use established tools under the hood, but I think you should take another look: - It manages the portable python version downloads depending on what you've pinned in your pyproject or the standard version: https://github.com/indygreg/python-build-standalone https://github.com/indygreg/python-build-standalone - It uses Hatch, uv (or piptools), venv etc. - It has a sane set of commands and should work even if they swap out some of the underlying tools. - Its fast and has a great coverage of PEPs. I've used it now for a few versions and I think its the best meta-packaging/python management tool out there. I
chpatrick 2y ago
I really like Nix for Python, as long as the packages I need are already packaged. Otherwise I'll use pip and venv to try stuff out. Can't stand conda.
- mistahchris 2y ago
  If you haven't yet, check out https://devenv.sh https://devenv.sh (super powered nix shell and more). It's pretty nice for python packages and installs your requirements to a project local venv for you via whatever tool you want (pip, poetry, uv etc). I've been using it for a couple of years and it's super nice to be able to manage both python and "native" dependencies, and other non-python development tools all together. I used just nix and whatever python packages are already in nixpkgs for several projects. And that works really really well until you run into an issue with compatibility like I did. It seems to mostly happen when some extremely common tool like `awscli2` depends on a specific version of some package and so it's pinned.
jen20 2y ago
I've found Nix to be basically the be-all-and-end-all of making Python work reasonably. Mitchell Hashimoto put out a post about how to actually do that a while ago [1]. [1]: https://mitchellh.com/writing/nix-with-dockerfiles https://mitchellh.com/writing/nix-with-dockerfiles
aborsy 2y ago
I use Python but I’m not a professional Python programmer. I use pip and venv, with requirements.txt. It copies Python binary and some of the shared standard libraries from the system, installs the required packages in sites-packages, and it always works fine. Am I missing anything major by not using conda and Poetry?
- PufPufPuf 2y ago
  That's generally fine until the build breaks due to a transitive dependency several levels down releasing a breaking change. While requirements.txt may hold enough information to build the project now, it is not guaranteed to work in the future -- you would need lockfiles for that guarantee. I've seen projects sort of emulating this by using "requirements.in" to list direct program dependencies, and auto-generating "requirements.txt" using "pip freeze". But when you find yourself doing that, it's probably time to switch to Poetry.
  - aborsy 2y ago
    I thought pip freeze > requirements.txt lists direct dependencies, as well as the transitive dependencies, and includes exact version numbers. From system, it copies the relevant binaries. In the future, pip install requirements.txt will install the exact same set of packages at those pinned versions. Unless the system components drastically change, the build is deterministic and project should run in the future. Sure, in the future if you change the version number for one package, the build might break, but that’s expected. It looks like the lock files specify the dependency graph, hashes of each package, URLs, and metadata about the system, so they do better when a package is updated.
forgottofloss 2y ago
I made https://pip.wtf https://pip.wtf, which is a "god damn it, I'm doing this myself" alternative for single-file scripts that just need some basic deps. You paste some code into your script and then it installs dependencies to a local directory.
- rangerelf 2y ago
  Ha! Genius, I love it!
- alright2565 2y ago
  Nice! The only criticism I have of this is needing to figure out the obscure shell quoting rules. But it is pretty easy to follow the DIY spirit and replace the os.system call with subprocess!
exe34 2y ago
What we need is a new package manager for Python, incorporating the lessons learnt from the others so far [0]. [0] https://xkcd.com/927/ https://xkcd.com/927/
- WillAdams 2y ago
  c.f., https://xkcd.com/1987/ https://xkcd.com/1987/
korijn 2y ago
I have worked with poetry professionally for about 5 years now and I am not looking back. It is exceptionally good. Dependency resolution speed is not an issue beyond the first run since all that hard to acquire metadata is actually cached in a local index. And even that first run is not particularly slow - _unless_ you depend on packages that are not available as wheels, which last I checked is not nearly as common nowadays as it was 10 years ago. However it can still happen: for example, if you are working with python 3.8 and you are using the latest version of some fancy library, they may have already stopped building wheels for that version of python. That means the package manager has to fall back to the sdist, and actually run the build scripts to acquire the metadata. On top of all this, private package feeds (like the one provided by azure devops) sometimes don't provide a metadata API at all, meaning the package manager has to download every single package just to get the metadata. The important bit of my little wall of text here though is that this is all true for all the other package managers as well. You can't necessarily attribute slow dependency resolution to a solver being written in C++ or pure python, given all of these other compounding factors which are often overlooked.
- odie5533 2y ago
  Try uv. You will be shocked how much faster it is.
  - korijn 2y ago
    I will! I'm sure it's faster when the data is available. But when it's not, in the common circumstances described above, network and disk IO are still the same unchanged bottlenecks, for any package manager. In conversations like this, we are all too quick to project our experiences on the package managers and not sharing in what circumstances we are using them.
mherrmann 2y ago
In 10 years of using Python mostly full-time, I have not yet had a project where pip and vent were not enough.
g8oz 2y ago
I just want to point out that Composer for PHP is a great experience.
- alganet 2y ago
  Composer did the right thing: it doesn't handle binaries, SAT for solving a _single_ version for the entire project tree and overall focus on simplicity. You need to handle PHP extensions outside of it. Composer can only warn you if your PHP lacks some compiled part, but won't act on it. It's good separation of concerns (and phpize/pecl already handles extensions very well). You also don't have multiple copies of the same package at different versions. Composer requires that a single version must be compatible. This sounds like a nightmare that would break easy, but it actually works by forcing package providers to be more backwards-compatible. That being said, we often don't use Composer to install tools in a machine, like we do with Python. Composer is there mostly to build standalone projects to be deployed elsewhere. You can `composer global install` stuff, but you often don't need to. I absolutely see this as a win. Tools for my machine should be installed by my OS package manager, not some language specific thing.
SirMaster 2y ago
This is what I never liked about python. How much work it could be to get some random python program from GitHub up and running. Compared to .NET and when I compile a framework-independent, single file executable.
yegle 2y ago
Why vendoring is not a common practice baffles me, especially since the leftpad incident happened over 8 years ago. For Python, you can use `pip wheel` (https://pip.pypa.io/en/stable/cli/pip_wheel/ https://pip.pypa.io/en/stable/cli/pip_wheel/) to download .whl files of your dependencies in a folder, add that folder to your version control, and update `sys.path` to include your .whl. For updating packages, you run `pip wheel` again and check in the new .whl files after carefully review the changes.
woodruffw 2y ago
> Naturally this led to a proliferation of new Python package managers which leverage the new standard. Enter poetry, PDM, Flit, and Hatch. An important qualification: Poetry uses pyproject.toml, but it doesn't use the standard (i.e. PEP 518, and 621) metadata layout. This in practice means that it doesn't follow the standard; it just happens to (confusingly) use a file with the same name. To the best of my knowledge, the others fully comply with the listed PEPs. In practice this means that the difference between them is abstracted away by tools like `build`, thanks to PEP 517.
simonw 2y ago
On Rye: "This project was ultimately abandoned by its author in 2023 and given to Astral.sh in favor of supporting uv instead" I don't think that's quite the right way to frame this. Handing Rye over to a company that could maintain it full time isn't the same thing as "abandoning" it - and the new maintainers are active on that project: https://github.com/astral-sh/rye/commits/main/ https://github.com/astral-sh/rye/commits/main/
- LarsDu88 2y ago
  Made a mistake here and have since corrected it
beaugunderson 2y ago
I share the author's excitement about `uv`. I was a big fan of `pip-compile` because it was the simplest possible way to have clear top-level dependencies that also froze sub-dependencies (you note your top-level dependencies in `requirements.in`, then use `pip-compile` to freeze those plus the sub-dependencies in `requirements.txt`, and it adds comments noting what top-level dependency brought it in). `uv` is basically that but faster.
poikroequ 2y ago
Poetry isn't perfect but I'm happy to have it. It installs most packages without issue and effortlessly handles multiple versions of Python on the same system. In my experience, Poetry works much better than, say... ... npm.
- lgas 2y ago
  ...which is why javascript has too many package managers too :)
- mountainriver 2y ago
  I like poetry but I’m not sure the Python ecosystem is disciplined enough for it, all too often I get stuck trying to resolve dependencies between packages which should be compatible but the dependencies weren’t defined well
- MrJohz 2y ago
  Can you describe what issues you've had with NPM that you haven't had with Poetry? In principle, they should be doing the same thing - both use a per-package environment and a lockfile, allow you to run commands in the environment, ensure the checked-in dependencies remain in sync with the local dependencies, etc. The main difference is that Python's ecosystem only allows one version of a dependency to exist at a time, but Node's ecosystem can handle multiple coexisting dependencies.
nrclark 2y ago
I ship a lot of Python in a CI/CD or devops context, and also deploy it to embedded targets. I've never needed anything more than pip, a venv, and pip-tools (to provide pip-compile). Venvs are treated as disposable. The basic workflow I use is: One-time (or as-needed for manual upgrades): 1. Make a venv with setuptools, wheel, and pip-tools (to get pip-compile) installed. 2. Use venv's pip-compile to generate a fully-pinned piptools-requirements.txt for the venv. 3. Check piptools-requirements.txt into my repo. This is used to get a stable, versioned `pip-compile` for use on my payload requirements. During normal development: 1. Add high-level dependencies to a `requirements.in` file. Usually unversioned, unless there's a good reason to specify something more exact. 2. On changes to `requirements.in`, make a venv from `piptools-requirements.txt` and its `pip-compile` to solve `requirements.in` into a fully-pinned `requirements.txt`. 3. Check requirements.in and requirements.txt into the repo. 4. Install packages from requirements.txt when making the venv that I need for production. This approach is very easy to automate, CI/CD friendly, and completely repeatable. It doesn't require any nonstandard tools to deploy (and only needs pip-compile when recompiling requirements.txt). It also makes a clear distinction between "what packages do the developers actually want?" and "what is the fully-versioned set of all dependencies". It's worked great for me over the years, and I'd highly recommend it as a reliable way to use the standard Python package tooling.
- sealedservant 2y ago
  My problem with pip-tools is this: # pip-tools dependencies dependencies = [ # direct dependencies "build >= 1.0.0", "click >= 8", "pip >= 22.2", "pyproject_hooks", "tomli; python_version < '3.11'", # indirect dependencies "setuptools", # typically needed when pip-tools invokes setup.py "wheel", # pip plugin needed by pip-tools ] pip is nice since it comes out of the box with Python. setuptools used to but now it's gone. The dependency explosion is a huge problem for a lot of people trying to lockdown the security and maintainability of their codebases. I think that's why a lot of people are rallying around Astral's projects like uv and ruff... they do so many things and they do them well.
  - nrclark 2y ago
artexy 2y ago
The people who say "just use pip and venv" don't understand the issue. Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. Would you like to copy external data into the staging area while using flit? You are out of luck, use poetry. Poetry versions <= 1.2.3.4.5.111 on the other hand do not support a backend. Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment. Which is the officially supported tool that will stay supported and not ripped out like distutils? Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks.
- JackSlateur 2y ago
  And reading your post, I still do not understand your issue. Is this because you can do various stuff in various way ? Would you like a single path, which must be followed ?
- eesmith 2y ago
  And if you want to build a Python/C extension, perhaps with a codegen step as well, then most of these modern tools throw their hands in the air. I had to hack my setup.py because PEP 517's fascination for completely isolated environment mean even a single typo fix in a Python comment meant a full re-compile of all 500KLOC in my main Python/C extension. Setuptools creates temporary directories for the .o files, and passes them to the build command. I had to override that to revert to the old behavior: class build_subclass(build): def finalize_options(self): self.build_lib = self.build_temp = self.build_platlib = None super().finalize_options() You might like my minimal wheel builder at https://news.ycombinator.com/item?id=38111355 https://news.ycombinator.com/item?id=38111355 . No dependencies, no framework.
- oblvious-earth 2y ago
  > The people who say "just use pip and venv" don't understand the issue. The people saying that say that because it works for them and it solved their problems. > setuptools is somewhat deprecated but not really setuptools remains the default and most popular backend (the thing that builds your package). What was deprecated was calling it directly, instead you use pip or build or any other frontend. Why? One reason is because it exposed lots of non-standard eccentricities that users would then start to rely on, breaking the effort of standardisation. > Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. They all use toml syntax, they all use the same build dependency standard, poetry does not use PEP 440 configuring version dependencies, but it does read PEP 440 dependencies from wheel metadata fine. I hope one day the version dependency spec can be updated with everything that has been learnt, from both Poetry and the rest of the ecosystem, but Python packaging is a slow moving beast. > Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment. pip does this for you though, I'm not sure why most end users need to know "build" or "install" exist. > Which is the officially supported tool that will stay supported and not ripped out like distutils? I'm not sure why you think any of these are "the officially supported tool", they are an attempt by various PyPA members to minimally and correctly implement the standards, but PyPA is a loose collection of volunteers. Whereas, CPython core dev have a pretty clear arms length attitude towards all of packaging. This for me feels like the real contention in the Python packaging world. > Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks. Well the people using pip and venv didn't have any of these questions, they just got on with it because it solved everything for them. Likely you want your package to figure something out as it’s being built, or have interesting metadata, or include binaries, or something else not on the happy path, and you want your build tool to "just work" even though chances are you’re doing something not that simple or what the tool creator thought. Otherwise, if you really just need a “simple zip archive of a directory” just write out the directory and call python -m zipfile. There’s nothing wrong with lots of people needing specific use cases, but it does mean all these build tools need to add lots and lots of options to accommodate them all, and before you know it you're complaining of a "gigantic framework" (even though I wouldn't describe either setuptools or hatchling like that).
tandav 2y ago
There is also virtualenvwrapper. It’s quite handy to create, list, remove virtual environment. I prefer to store all venvs in ~/.cache/virtualenvs instead of .venv in project directory, makes it more clean, no need to exclude for backups or git repository. https://virtualenvwrapper.readthedocs.io/en/latest/ https://virtualenvwrapper.readthedocs.io/en/latest/
VeejayRampay 2y ago
can't wait to throw all that mediocre tooling away over when astral has finally nailed it like we all know they will
jacobgorm 2y ago
Cargo is strictly worse than any of the solutions for python. Doing even the simplest things seems to involve pulling down and compiling hundreds of dependencies.
- big-green-man 2y ago
  This is... Not a bad thing? Compiling your dependencies into your standalone executable is like butter, no dependency hell, no virtual environments or containers to avoid said hell. You release a program that has everything it needs to run, it's as good as it gets IMO.
  - jacobgorm 2y ago
    The problem that the resulting program ends up too large for my users to download, because it depended on libtwoplustwo that in turn depended on libkitchensink.
    - big-green-man 2y ago
      The rust compiler does not include in your binary any unused code. Unless libtwoplustwo uses the entirety of libkitchensink, the latter is not entirely included in your binary. This is not true for python, or external libraries, or appimages, or docker containers.
      - jacobgorm 2y ago
        It seems to me one would have to solve the Halting Problem for that to be universally true.
        big-green-man 2y ago
        Not quite, it is pretty trivial to check what external dependencies your program calls, and what those dependencies call and so forth. I'm sure you could write a program that has a dependency chain that requires the entire database of packages in cargo (and that would he an interesting experiment). A lot of modern programming languages evaluate what code is referenced in a working program and only include used code, and the compiler enforces writing your program in such a way that it doesn't have to go over infinite possibilities to check. Of course, you have to have the entire dependency to compile for each import (and so would your user if you only distribute source code) to reference the first problem you mentioned above, and I'm sure there are ways to trick the compiler into endlessly checking dependencies or evaluating references, probably using unsafe, but that's why the compiler won't let you compile with unassigned values and has ownership and scope, you have to write your program in such a way that the compiler can tell what code will be executed at runtime or it won't compile.
joeyagreco 2y ago
1. Use pip as a package manager 2. If you need a fresh env to test in, use venv This is the flow I follow for Python development
big-green-man 2y ago
I think this problem comes from ideas we had before that were well intentioned. The idea that led to all of this is, what if we could save developers a ton of work and users a ton of disk space by packaging libraries for them to use? And then we have a ton of problems as a result of this decision. C is now your system's API, mismatched dependencies, a gazillion package managers. And then what solution do we have? Virtual environments, virtual machines, docker and appimage. We package all dependencies and even entire operating systems so as to avoid all these problems. It's legacy support all the way down. From scratch, I'd say, devs should just pull all dependencies into their code and package them with their product. Users should never even have to touch something like pip, or a virtual environment. A package manager that allows a developer to publish tools others can use to build code, but that packages the dependencies with their package instead of pulling it for users, would be ideal. Where possible, avoid dependency on anything external entirely. What's that XKCD about yet another standard? I know it will never happen, but I sure do wish it worked like that.
VGHN7XDuOXPAzol 2y ago
when we did a comparison of package managers that lock dependencies, we wrote up some interesting notes at https://github.com/matrix-org/synapse/issues/11537#issuecomment-1015975819 https://github.com/matrix-org/synapse/issues/11537#issuecomm... Notable omission in pip-tools which many are suggesting here as being simpler: it can't write requirements files for multiple environments/platforms without running it once for each of those environments and having one file for all of them. We settled on Poetry at the time but it has been quite unstable overall. Not so bad recently but there were a lot of issues/regressions with it over time. For this reason I am happy to see new takes on package management, hopefully some of these will have clearer wins over the others, where you have to spend ages trying to figure out which one will do what you need.
dogaar 2y ago
One thing this blog does not question is whether package managers are needed at all. Look at how Deno does it: packages are imported from URLs and downloaded to a local cache at run-time. This allows CDN-based package distribution, but you still need a package search engine.

131 points by LarsDu88 2y ago

Python actually now has over a dozen package managers, none of which are as good as Cargo in Rust. Here's my attempt at a nearly comprehensive rundown.

eesmith 2y ago
"like PyPy" -> "like PyPI".
- LarsDu88 2y ago
  fixed

foundart 2y ago

Package management has been a problem almost every time I have dabbled in Python. This is a great overview of the situation and will save me time the next time.

big-green-man 2y ago
It didn't used to be like this. I remember first starting in Python, around a decade ago, when Python3 has been out a long time but everyone still wrote in Python2 for some reason. You used pip, that's it. Everything was easy. There were very good reasons to make some of the changes that have been made, but I think that big switch kind of normalized breaking changes in the python world. At this point we have a catastrophe on our hands. A language and it's tooling is supposed to get out of my way, it's supposed to be the tool by which I express my intention, not be a thing I have to tinker with all day. There's room for being opinionated, and there's room for upgrading things, but if I need to follow 10 RSS feeds just to keep up to date with changes, if I'm arguing with my colleagues about which of a dozen ways to use a language is best, something has gone horribly wrong.

odie5533 2y ago

I disagree with the title, but it's an okay rundown of the various package managers. Wish the author had tried out hatch though since it seems good. Also rye is not abandoned. You can see the repo has updates within the last 24 hours for a new release. I think they want uv to eat rye, but that hasn't happened yet. My current favorites are uv + mise. Handles lockfiles, multiple versions of python, and it's very fast since uv is very fast. Have not tried pdm or hatch though.

PaulHoule 2y ago
uv is transformative because it is (1) correct and (2) crazy fast. I wanted to like Poetry but the performance is atrocious. On the other hand, with uv, I feel like I can always build a new instance of a system whenever I want, whereas with conda or poetry I might have to wait ten minutes.
- heyoni 2y ago
  I think the company that makes UV hired the guy that created ripgrep so no wonder it’s so fast (burntsushi i think is his alias)
deleted 2y ago
[deleted]
alanwreath 2y ago
you mean it is in the process of happening? From this article it looks like Armin R. and Charlie M. are updating Rye use uv under the hood. https://astral.sh/blog/uv https://astral.sh/blog/uv > As part of this release, we're also taking stewardship of Rye, an experimental Python packaging tool from Armin Ronacher. We'll maintain Rye as we expand uv into a unified successor project, to fulfill our shared vision for Python packaging.

gjvc 2y ago

usual bullshite about pip and venv by people who think the "activate" script does something useful

igorguerrero 2y ago

Someone from Rust talking about these tools being slow... That's rich.

deleted 2y ago
[deleted]

actionfromafar 2y ago

After all, the motto is "there's more than one way to do it"!

rilindo 2y ago
That is actually Perl's motto. [0] [0] https://wiki.c2.com/?ThereIsMoreThanOneWayToDoIt https://wiki.c2.com/?ThereIsMoreThanOneWayToDoIt
- binary132 2y ago
  I think that’s the joke
- drewcoo 2y ago
  But Perl started this whole one package manager to bind us thing with CPAN.
theideaofcoffee 2y ago
Maybe it's just a joke, but for those that don't know that is Perl's motto and even better because there's the venerable, wonderful CPAN [0] which is the de facto package repository for the language. [0] https://www.cpan.org/ https://www.cpan.org/
- deleted 2y ago
  [deleted]
- marcosdumay 2y ago
  It's certainly a joke, because the Python original motto was "there's only one way to do it", and the current one is "there's only one (obvious) way to do it". The Python's motto was created as an obvious reference to the Perl's one, purposefully negating it.
  - theideaofcoffee 2y ago
    Thank you for that context!

AtlasBarfed 2y ago

"Finally, because dependency resolution is a directed acylic graph (DAG)" ... Is it? :-) Mwahahaha

glitchc 2y ago
They forgot the word "ideally"
stairlane 2y ago
We analyze firmware and binaries, amongst other things at my day job. Reconstructing dependency graphs is a necessary part of our day to day. Assuming dependency graphs cant have cycles has shot us in the foot. There are absolutely real circular dependencies in assets in the wild.
- LarsDu88 2y ago
  Original author here... wait, what? I believe pip (or at least older versions of pip) makes it possible to have cycles. Are you analyzing dep graphs made with older versions of pip?
  - stairlane 2y ago
    Not pip, but we ran into an older version of glibc that had real circular deps. Sorry I don’t remember off the top of my head which one.
rzzzt 2y ago
"Hey Vsauce, Michael here." https://youtu.be/TN25ghkfgQA https://youtu.be/TN25ghkfgQA

sealedservant 2y ago

I concur and I also think that there are too many build backends. pdm is my current favorite package manager. It is fully PEP-compliant and the lockfile generation is nice. I wouldn't call hatch a package manager because I don't think it can make lockfiles. uv is on my radar but it doesn't look ready for primetime yet. I saw they are building out a package management API with commands such as `uv add` and `uv remove`. Cross-platform lockfiles, editable installs, in-project .venv, and a baked-in build backend might be enough for me to make the switch. It's my pipe dream to get the full build/install/publish workflow down to a single static binary with no dependencies. Anna-Lee Popkes has an awesome comparison of the available package managers [0] complete with handy Venn diagrams. The pyOpenSci team has another nice comparison of the available build backends [1]. [0] https://alpopkes.com/posts/python/packaging_tools/ https://alpopkes.com/posts/python/packaging_tools/ [1] https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html https://www.pyopensci.org/python-package-guide/package-struc...

PrivateButts 2y ago
pdm is my go to as well. I've decided to really jump on the pyproject.toml train and pdm plays very well with it. .venv by default is pretty nice as well. Anything, and I mean anything, is better than pipenv though.
mih 2y ago
Another user of pdm here for professional projects. It sure is more standards compliant than poetry. Support for in-project venvs and integration of configs for packages such as pytest is quite useful. When evaluating package managers, poetry for sure was a contender. However listening to others experiences regarding poetry developers introducing breaking changes that could potentially cause the CI pipeline made it a no go [1]. uv seems to be coming along rather nicely, but wasn't anywhere near the level of stability compared to pdm during the evaluation phase. [1] https://www.youtube.com/watch?v=Gr9o8MW_pb0 https://www.youtube.com/watch?v=Gr9o8MW_pb0
- odie5533 2y ago
  Poetry supports in-project venvs. You can enable it globally. https://python-poetry.org/docs/configuration/#virtualenvsin-project https://python-poetry.org/docs/configuration/#virtualenvsin-...

p1esk 2y ago

Looks like Conda is still the best package/env manager for ML engineers.

dmd 2y ago
Unfortunately I'm still waiting for "Solving environment". It's been 8 years. Please send help. I'm so hungry. (Yes, I know about mamba.)
- LarsDu88 2y ago
  I lol'd at this. At my last job, we lost days if not weeks to "solving environment" due to poetry and many folks not understanding that ceiling pinning is BAD when you use a tool like poetry to manage your deps.
hrpnk 2y ago
Poetry user here. What is so special about Conda that makes it particularly good for ML use cases?
- p1esk 2y ago
  No idea, I just summarized TFA.
gillesjacobs 2y ago
ML engineer, I am up to 3 irreversibly broken environments after simply adding a package. Conda is inexcusably slow and but even mamba can save it. Every conda project I have ever was later switched to poetry. Here's a fun challenge: try to determine channel priority in your conda env, go ahead try. Conda is the single worst packaging tool I have ever used.
- p1esk 2y ago
  OK, would you say poetry is the best for ML engineers?

wantsanagent 2y ago

I use pip. I plan to continue using pip. If I need an isolated environment, I use conda, but then I install everything with pip. If I need to guarantee versions I pip freeze. There's a lot of cruft and desire for a one-size-fits-all solution but the base tools are probably good enough. My setup is not the one-size-fits-all solution but it works for me, and my team, and lots of other teams. Beware anyone who tells you that thirty years of tooling doesn't have a solution to the problem you're facing and you need this new shiny thing.* *Playing with shiny things is fun and should not be discouraged, but must not be mandated either.

colemannerd 2y ago
BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.
- yjftsjthsd-h 2y ago
  > For me, this setup has caused production outages multiple times across multiple teams How does that happen?
  - colemannerd 2y ago
    I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.
    - yjftsjthsd-h 2y ago
      The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?

slt2021 2y ago

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need. This is similar approach to “zero-based budgeting”. It forces you to carefully pick your dependencies and think about what you carry. I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade If you dont do that, you can quickly go down the javascript’s path of bloated node_modules. Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? Even venv is not really required if you dockerize your python apps, which you will have to do anyways at deploy time

BerislavLopac 2y ago
> Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? It's very simple: 1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.
krapht 2y ago
Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems. Conda solves this pain point by being a general purpose package manager. Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.
sealedservant 2y ago
Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.

SethMLarson 2y ago

Happily using pip, venv, and pip-tools for every project and still finding them more than suitable. They might not have the marketing budget or pizazz of others, but if you're looking for effective and boring tools that get the job done so you can solve more interesting problems they work just fine.

verst 2y ago
Same. Additionally I use pyenv and pyenv-virtualenv to manage multiple Python versions. Different venvs pointing to different Python versions etc, but the core tools are still pip, venv and pip-tools. (I do not use conda) I publish a Python SDK with about 85k monthly downloads according to PyPI. I make sure to run tests (unit tests, type checking, integration tests) against all currently supported minor versions of Python (3.8 - 3.12)
- epistasis 2y ago
  I think pyenv finally solved the versioning problem for me. Additionally, being able to set a Python version or virtual environment per-directory with "pyenv local" which has eliminated having to remember which venv I was using on a project directory, or remembering which convention for venvs I used for a project.
  - verst 2y ago
    And "pyenv shell" is also handy for temporarily changing the default Python version of your current shell session. Or changing the global default using "pyenv global" - for example I currently have this set to "3.12.3", even though I usually test the SDKs I build against the oldest version I have to support first - the latest 3.8.X

VagabundoP 2y ago

Use Rye. It wasn't abandoned it ownership was transferred. Rye uses other pretty standard stuff under the hood, tools that follow PEPs, its just a front end that is sane. uv is fast as well. It downloads the pinned version of standalone Python, it keeps everything in its own venv and theres very little messing/tweaking of the environment. It is messy, although its getting better. I doubt everything will ever standardise to one tool however.

LarsDu88 2y ago
I made a mistake about Rye when I posted the first draft of this article. That has since been corrected. Rye is incredibly young, and when I first encountered it, I looked at its source code and found it was simply calling piptools under the hood. I did not realize that it had changed and now uses uv (or perhaps just the same solver as uv) under the hood.
- VagabundoP 2y ago
  It is young, and it does use established tools under the hood, but I think you should take another look: - It manages the portable python version downloads depending on what you've pinned in your pyproject or the standard version: https://github.com/indygreg/python-build-standalone https://github.com/indygreg/python-build-standalone - It uses Hatch, uv (or piptools), venv etc. - It has a sane set of commands and should work even if they swap out some of the underlying tools. - Its fast and has a great coverage of PEPs. I've used it now for a few versions and I think its the best meta-packaging/python management tool out there. I

chpatrick 2y ago

I really like Nix for Python, as long as the packages I need are already packaged. Otherwise I'll use pip and venv to try stuff out. Can't stand conda.

mistahchris 2y ago
If you haven't yet, check out https://devenv.sh https://devenv.sh (super powered nix shell and more). It's pretty nice for python packages and installs your requirements to a project local venv for you via whatever tool you want (pip, poetry, uv etc). I've been using it for a couple of years and it's super nice to be able to manage both python and "native" dependencies, and other non-python development tools all together. I used just nix and whatever python packages are already in nixpkgs for several projects. And that works really really well until you run into an issue with compatibility like I did. It seems to mostly happen when some extremely common tool like `awscli2` depends on a specific version of some package and so it's pinned.

jen20 2y ago

I've found Nix to be basically the be-all-and-end-all of making Python work reasonably. Mitchell Hashimoto put out a post about how to actually do that a while ago [1]. [1]: https://mitchellh.com/writing/nix-with-dockerfiles https://mitchellh.com/writing/nix-with-dockerfiles

aborsy 2y ago

I use Python but I’m not a professional Python programmer. I use pip and venv, with requirements.txt. It copies Python binary and some of the shared standard libraries from the system, installs the required packages in sites-packages, and it always works fine. Am I missing anything major by not using conda and Poetry?

PufPufPuf 2y ago
That's generally fine until the build breaks due to a transitive dependency several levels down releasing a breaking change. While requirements.txt may hold enough information to build the project now, it is not guaranteed to work in the future -- you would need lockfiles for that guarantee. I've seen projects sort of emulating this by using "requirements.in" to list direct program dependencies, and auto-generating "requirements.txt" using "pip freeze". But when you find yourself doing that, it's probably time to switch to Poetry.
- aborsy 2y ago
  I thought pip freeze > requirements.txt lists direct dependencies, as well as the transitive dependencies, and includes exact version numbers. From system, it copies the relevant binaries. In the future, pip install requirements.txt will install the exact same set of packages at those pinned versions. Unless the system components drastically change, the build is deterministic and project should run in the future. Sure, in the future if you change the version number for one package, the build might break, but that’s expected. It looks like the lock files specify the dependency graph, hashes of each package, URLs, and metadata about the system, so they do better when a package is updated.

forgottofloss 2y ago

I made https://pip.wtf https://pip.wtf, which is a "god damn it, I'm doing this myself" alternative for single-file scripts that just need some basic deps. You paste some code into your script and then it installs dependencies to a local directory.

rangerelf 2y ago
Ha! Genius, I love it!
alright2565 2y ago
Nice! The only criticism I have of this is needing to figure out the obscure shell quoting rules. But it is pretty easy to follow the DIY spirit and replace the os.system call with subprocess!

exe34 2y ago

What we need is a new package manager for Python, incorporating the lessons learnt from the others so far [0]. [0] https://xkcd.com/927/ https://xkcd.com/927/

WillAdams 2y ago
c.f., https://xkcd.com/1987/ https://xkcd.com/1987/

korijn 2y ago

I have worked with poetry professionally for about 5 years now and I am not looking back. It is exceptionally good. Dependency resolution speed is not an issue beyond the first run since all that hard to acquire metadata is actually cached in a local index. And even that first run is not particularly slow - _unless_ you depend on packages that are not available as wheels, which last I checked is not nearly as common nowadays as it was 10 years ago. However it can still happen: for example, if you are working with python 3.8 and you are using the latest version of some fancy library, they may have already stopped building wheels for that version of python. That means the package manager has to fall back to the sdist, and actually run the build scripts to acquire the metadata. On top of all this, private package feeds (like the one provided by azure devops) sometimes don't provide a metadata API at all, meaning the package manager has to download every single package just to get the metadata. The important bit of my little wall of text here though is that this is all true for all the other package managers as well. You can't necessarily attribute slow dependency resolution to a solver being written in C++ or pure python, given all of these other compounding factors which are often overlooked.

odie5533 2y ago
Try uv. You will be shocked how much faster it is.
- korijn 2y ago
  I will! I'm sure it's faster when the data is available. But when it's not, in the common circumstances described above, network and disk IO are still the same unchanged bottlenecks, for any package manager. In conversations like this, we are all too quick to project our experiences on the package managers and not sharing in what circumstances we are using them.

mherrmann 2y ago

In 10 years of using Python mostly full-time, I have not yet had a project where pip and vent were not enough.

g8oz 2y ago

I just want to point out that Composer for PHP is a great experience.

alganet 2y ago
Composer did the right thing: it doesn't handle binaries, SAT for solving a _single_ version for the entire project tree and overall focus on simplicity. You need to handle PHP extensions outside of it. Composer can only warn you if your PHP lacks some compiled part, but won't act on it. It's good separation of concerns (and phpize/pecl already handles extensions very well). You also don't have multiple copies of the same package at different versions. Composer requires that a single version must be compatible. This sounds like a nightmare that would break easy, but it actually works by forcing package providers to be more backwards-compatible. That being said, we often don't use Composer to install tools in a machine, like we do with Python. Composer is there mostly to build standalone projects to be deployed elsewhere. You can `composer global install` stuff, but you often don't need to. I absolutely see this as a win. Tools for my machine should be installed by my OS package manager, not some language specific thing.

SirMaster 2y ago

This is what I never liked about python. How much work it could be to get some random python program from GitHub up and running. Compared to .NET and when I compile a framework-independent, single file executable.

yegle 2y ago

Why vendoring is not a common practice baffles me, especially since the leftpad incident happened over 8 years ago. For Python, you can use `pip wheel` (https://pip.pypa.io/en/stable/cli/pip_wheel/ https://pip.pypa.io/en/stable/cli/pip_wheel/) to download .whl files of your dependencies in a folder, add that folder to your version control, and update `sys.path` to include your .whl. For updating packages, you run `pip wheel` again and check in the new .whl files after carefully review the changes.

woodruffw 2y ago

> Naturally this led to a proliferation of new Python package managers which leverage the new standard. Enter poetry, PDM, Flit, and Hatch. An important qualification: Poetry uses pyproject.toml, but it doesn't use the standard (i.e. PEP 518, and 621) metadata layout. This in practice means that it doesn't follow the standard; it just happens to (confusingly) use a file with the same name. To the best of my knowledge, the others fully comply with the listed PEPs. In practice this means that the difference between them is abstracted away by tools like `build`, thanks to PEP 517.

simonw 2y ago

On Rye: "This project was ultimately abandoned by its author in 2023 and given to Astral.sh in favor of supporting uv instead" I don't think that's quite the right way to frame this. Handing Rye over to a company that could maintain it full time isn't the same thing as "abandoning" it - and the new maintainers are active on that project: https://github.com/astral-sh/rye/commits/main/ https://github.com/astral-sh/rye/commits/main/

LarsDu88 2y ago
Made a mistake here and have since corrected it

beaugunderson 2y ago

I share the author's excitement about `uv`. I was a big fan of `pip-compile` because it was the simplest possible way to have clear top-level dependencies that also froze sub-dependencies (you note your top-level dependencies in `requirements.in`, then use `pip-compile` to freeze those plus the sub-dependencies in `requirements.txt`, and it adds comments noting what top-level dependency brought it in). `uv` is basically that but faster.

poikroequ 2y ago

Poetry isn't perfect but I'm happy to have it. It installs most packages without issue and effortlessly handles multiple versions of Python on the same system. In my experience, Poetry works much better than, say... ... npm.

lgas 2y ago
...which is why javascript has too many package managers too :)
mountainriver 2y ago
I like poetry but I’m not sure the Python ecosystem is disciplined enough for it, all too often I get stuck trying to resolve dependencies between packages which should be compatible but the dependencies weren’t defined well
MrJohz 2y ago
Can you describe what issues you've had with NPM that you haven't had with Poetry? In principle, they should be doing the same thing - both use a per-package environment and a lockfile, allow you to run commands in the environment, ensure the checked-in dependencies remain in sync with the local dependencies, etc. The main difference is that Python's ecosystem only allows one version of a dependency to exist at a time, but Node's ecosystem can handle multiple coexisting dependencies.

nrclark 2y ago

I ship a lot of Python in a CI/CD or devops context, and also deploy it to embedded targets. I've never needed anything more than pip, a venv, and pip-tools (to provide pip-compile). Venvs are treated as disposable. The basic workflow I use is: One-time (or as-needed for manual upgrades): 1. Make a venv with setuptools, wheel, and pip-tools (to get pip-compile) installed. 2. Use venv's pip-compile to generate a fully-pinned piptools-requirements.txt for the venv. 3. Check piptools-requirements.txt into my repo. This is used to get a stable, versioned `pip-compile` for use on my payload requirements. During normal development: 1. Add high-level dependencies to a `requirements.in` file. Usually unversioned, unless there's a good reason to specify something more exact. 2. On changes to `requirements.in`, make a venv from `piptools-requirements.txt` and its `pip-compile` to solve `requirements.in` into a fully-pinned `requirements.txt`. 3. Check requirements.in and requirements.txt into the repo. 4. Install packages from requirements.txt when making the venv that I need for production. This approach is very easy to automate, CI/CD friendly, and completely repeatable. It doesn't require any nonstandard tools to deploy (and only needs pip-compile when recompiling requirements.txt). It also makes a clear distinction between "what packages do the developers actually want?" and "what is the fully-versioned set of all dependencies". It's worked great for me over the years, and I'd highly recommend it as a reliable way to use the standard Python package tooling.

sealedservant 2y ago
My problem with pip-tools is this: # pip-tools dependencies dependencies = [ # direct dependencies "build >= 1.0.0", "click >= 8", "pip >= 22.2", "pyproject_hooks", "tomli; python_version < '3.11'", # indirect dependencies "setuptools", # typically needed when pip-tools invokes setup.py "wheel", # pip plugin needed by pip-tools ] pip is nice since it comes out of the box with Python. setuptools used to but now it's gone. The dependency explosion is a huge problem for a lot of people trying to lockdown the security and maintainability of their codebases. I think that's why a lot of people are rallying around Astral's projects like uv and ruff... they do so many things and they do them well.
- nrclark 2y ago

artexy 2y ago

The people who say "just use pip and venv" don't understand the issue. Distutils has been ripped out of Python core, setuptools is somewhat deprecated but not really. Just don't call setup.py directly. Or use flit. Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. Would you like to copy external data into the staging area while using flit? You are out of luck, use poetry. Poetry versions <= 1.2.3.4.5.111 on the other hand do not support a backend. Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment. Which is the officially supported tool that will stay supported and not ripped out like distutils? Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks.

JackSlateur 2y ago
And reading your post, I still do not understand your issue. Is this because you can do various stuff in various way ? Would you like a single path, which must be followed ?
eesmith 2y ago
And if you want to build a Python/C extension, perhaps with a codegen step as well, then most of these modern tools throw their hands in the air. I had to hack my setup.py because PEP 517's fascination for completely isolated environment mean even a single typo fix in a Python comment meant a full re-compile of all 500KLOC in my main Python/C extension. Setuptools creates temporary directories for the .o files, and passes them to the build command. I had to override that to revert to the old behavior: class build_subclass(build): def finalize_options(self): self.build_lib = self.build_temp = self.build_platlib = None super().finalize_options() You might like my minimal wheel builder at https://news.ycombinator.com/item?id=38111355 https://news.ycombinator.com/item?id=38111355 . No dependencies, no framework.
oblvious-earth 2y ago
> The people who say "just use pip and venv" don't understand the issue. The people saying that say that because it works for them and it solved their problems. > setuptools is somewhat deprecated but not really setuptools remains the default and most popular backend (the thing that builds your package). What was deprecated was calling it directly, instead you use pip or build or any other frontend. Why? One reason is because it exposed lots of non-standard eccentricities that users would then start to rely on, breaking the effort of standardisation. > Or perhaps pyproject.toml? If the latter, flit, poetry and the 100 other frontends all have a different syntax. They all use toml syntax, they all use the same build dependency standard, poetry does not use PEP 440 configuring version dependencies, but it does read PEP 440 dependencies from wheel metadata fine. I hope one day the version dependency spec can be updated with everything that has been learnt, from both Poetry and the rest of the ecosystem, but Python packaging is a slow moving beast. > Should you use pip? Well, the latest hotness are the search-unfriendly named modules "build" and "install", which create a completely isolated environment. pip does this for you though, I'm not sure why most end users need to know "build" or "install" exist. > Which is the officially supported tool that will stay supported and not ripped out like distutils? I'm not sure why you think any of these are "the officially supported tool", they are an attempt by various PyPA members to minimally and correctly implement the standards, but PyPA is a loose collection of volunteers. Whereas, CPython core dev have a pretty clear arms length attitude towards all of packaging. This for me feels like the real contention in the Python packaging world. > Questions over questions. And all that for creating a simple zip archive of a directory, which for some reason demands gigantic frameworks. Well the people using pip and venv didn't have any of these questions, they just got on with it because it solved everything for them. Likely you want your package to figure something out as it’s being built, or have interesting metadata, or include binaries, or something else not on the happy path, and you want your build tool to "just work" even though chances are you’re doing something not that simple or what the tool creator thought. Otherwise, if you really just need a “simple zip archive of a directory” just write out the directory and call python -m zipfile. There’s nothing wrong with lots of people needing specific use cases, but it does mean all these build tools need to add lots and lots of options to accommodate them all, and before you know it you're complaining of a "gigantic framework" (even though I wouldn't describe either setuptools or hatchling like that).

tandav 2y ago

There is also virtualenvwrapper. It’s quite handy to create, list, remove virtual environment. I prefer to store all venvs in ~/.cache/virtualenvs instead of .venv in project directory, makes it more clean, no need to exclude for backups or git repository. https://virtualenvwrapper.readthedocs.io/en/latest/ https://virtualenvwrapper.readthedocs.io/en/latest/

VeejayRampay 2y ago

can't wait to throw all that mediocre tooling away over when astral has finally nailed it like we all know they will

jacobgorm 2y ago

Cargo is strictly worse than any of the solutions for python. Doing even the simplest things seems to involve pulling down and compiling hundreds of dependencies.

big-green-man 2y ago
This is... Not a bad thing? Compiling your dependencies into your standalone executable is like butter, no dependency hell, no virtual environments or containers to avoid said hell. You release a program that has everything it needs to run, it's as good as it gets IMO.
- jacobgorm 2y ago
  The problem that the resulting program ends up too large for my users to download, because it depended on libtwoplustwo that in turn depended on libkitchensink.
  - big-green-man 2y ago
    The rust compiler does not include in your binary any unused code. Unless libtwoplustwo uses the entirety of libkitchensink, the latter is not entirely included in your binary. This is not true for python, or external libraries, or appimages, or docker containers.
    - jacobgorm 2y ago
      It seems to me one would have to solve the Halting Problem for that to be universally true.
      - big-green-man 2y ago
        Not quite, it is pretty trivial to check what external dependencies your program calls, and what those dependencies call and so forth. I'm sure you could write a program that has a dependency chain that requires the entire database of packages in cargo (and that would he an interesting experiment). A lot of modern programming languages evaluate what code is referenced in a working program and only include used code, and the compiler enforces writing your program in such a way that it doesn't have to go over infinite possibilities to check. Of course, you have to have the entire dependency to compile for each import (and so would your user if you only distribute source code) to reference the first problem you mentioned above, and I'm sure there are ways to trick the compiler into endlessly checking dependencies or evaluating references, probably using unsafe, but that's why the compiler won't let you compile with unassigned values and has ownership and scope, you have to write your program in such a way that the compiler can tell what code will be executed at runtime or it won't compile.

joeyagreco 2y ago

1. Use pip as a package manager 2. If you need a fresh env to test in, use venv This is the flow I follow for Python development

big-green-man 2y ago

I think this problem comes from ideas we had before that were well intentioned. The idea that led to all of this is, what if we could save developers a ton of work and users a ton of disk space by packaging libraries for them to use? And then we have a ton of problems as a result of this decision. C is now your system's API, mismatched dependencies, a gazillion package managers. And then what solution do we have? Virtual environments, virtual machines, docker and appimage. We package all dependencies and even entire operating systems so as to avoid all these problems. It's legacy support all the way down. From scratch, I'd say, devs should just pull all dependencies into their code and package them with their product. Users should never even have to touch something like pip, or a virtual environment. A package manager that allows a developer to publish tools others can use to build code, but that packages the dependencies with their package instead of pulling it for users, would be ideal. Where possible, avoid dependency on anything external entirely. What's that XKCD about yet another standard? I know it will never happen, but I sure do wish it worked like that.

VGHN7XDuOXPAzol 2y ago

when we did a comparison of package managers that lock dependencies, we wrote up some interesting notes at https://github.com/matrix-org/synapse/issues/11537#issuecomment-1015975819 https://github.com/matrix-org/synapse/issues/11537#issuecomm... Notable omission in pip-tools which many are suggesting here as being simpler: it can't write requirements files for multiple environments/platforms without running it once for each of those environments and having one file for all of them. We settled on Poetry at the time but it has been quite unstable overall. Not so bad recently but there were a lot of issues/regressions with it over time. For this reason I am happy to see new takes on package management, hopefully some of these will have clearer wins over the others, where you have to spend ages trying to figure out which one will do what you need.

dogaar 2y ago

One thing this blog does not question is whether package managers are needed at all. Look at how Deno does it: packages are imported from URLs and downloaded to a local cache at run-time. This allows CDN-based package distribution, but you still need a package search engine.