9 ms·

I never used anything other than pip. I never felt the need to use anything other than pip (with virtualenv). Am I missing anything?

mplewis 2y ago
Pip only has requirements.txt and doesn't have lockfiles, so you can't guarantee that the bugs you're seeing on your system are the same as the bugs on your production system.
- aidos 2y ago
  I’ve always worked around that by having a requirements.base.txt and a requirements.txt for the locked versions. Obviously pip doesn’t do that for you but it’s not hard to manage yourself. Having said that, I’m going to give uv a shot because I hear so many good things about it.
  - mikepurvis 2y ago
    I’m grouchy because I finally got religion on poetry a few years ago, but the hype on uv is good enough that I’ll have to give it a shot.
    - kstrauser 2y ago
      I freaking love Poetry. It was a huge breath of fresh air after years of pip and a short detour with Pipenv. If uv stopped existing I’d go back to Poetry. But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.
    - remram 2y ago
      With the new major release of Poetry that just came out I also feel like it might be a good time to switch to Uv rather than adapt to this new version: https://python-poetry.org/blog/announcing-poetry-2.0.0/ https://python-poetry.org/blog/announcing-poetry-2.0.0/
  - selcuka 2y ago
    With pip the best practice is to have a requirements.txt with direct requirements (strictly or loosely pinned), and a separate constraints.txt file [1] with strictly pinned versions of all direct- and sub-dependencies (basically the output of `pip freeze`). The latter works like a lock file. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
    - zahlman 2y ago
      For direct requirements you're better off using the `pyproject.toml` for direct dependencies (and you can plausibly use it to pin everything if you're developing an application). It's project metadata that you'll need anyway for building your project, and the "editable wheel" hack allows Pip to use that information to set up an environment for your code (via `pip install -e .`).
  - remram 2y ago
    This works until you need to upgrade something, pip might upgrade to a broken set of dependencies. Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.
    - zahlman 2y ago
      > pip might upgrade to a broken set of dependencies. I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end. > Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix. The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo: $ pip install 'numpy;python_version<="2.7"' Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment > There are a lot of gotchas that pip can't fix. There are a lot of serious problems with Pip - I just don't think these are among them.
      - remram 2y ago
        You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.
        zahlman 2y ago
        Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)
- zo1 2y ago
  The requirements.txt file is the lockfile. Anyways, this whole obsession with locked deps or "lockfiles" is such an anti-pattern, I have no idea why we went there as an industry. Probably as a result of some of the newer stuff that is classified as "hipster-tech" such as docker and javascript.
  - n144q 2y ago
    Just because you don't understand it, it's ok to call it an "anti-pattern"? Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you. If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.
    - driggs 2y ago
      Just because someone has a different perspective than you doesn't mean they don't "understand". Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.
      - Uvix 2y ago
        If you're developing a library, and you have a requirement for what's normally a transitive dependency, it should be specified as a top-level dependency.
        driggs 2y ago
        The point is that if I'm writing a library and I specify `requests == 1.2.3`, then what are you going to do in your application if you need both my library and `requests == 1.2.4`? This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible. It's the developers of an application who should use a lockfile to lock transitive dependencies.
- polski-g 2y ago
  "pip freeze" generates a lockfile.
  - orf 2y ago
    No, that generates a list of currently installed packages. That’s very much not a lock file, even if it is possible to abuse it as such.
    - deleted 2y ago
      [deleted]
    - zahlman 2y ago
      A list of currently installed packages in the current environment, with their exact versions. This is only the actually needed packages, with their transitive dependencies, unless you've left something behind from earlier in development. If you're keeping abstract dependencies up to date in `pyproject.toml` (which you need to do anyway to build and release the project), you can straightforwardly re-create the environment from that list and freeze whatever solution you get (after testing).
      - orf 2y ago
        Doesn’t account for differences in platforms or Python versions, and doesn’t contain resolved dependency hashes. So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.
        zahlman 2y ago
        Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution. >only works on the exact same Python version It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each. In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-a-tomlitomllib-compatibility-layer https://github.com/hukkin/tomli?tab=readme-ov-file#building-...). >on my machine (Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.) If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.) This is also only relevant for projects that include non-Python code that requires a build step, of course. >assuming no dependencies have published new packages. PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.
amluto 2y ago
If you switch to uv, you’ll have fewer excuses to take coffee breaks while waiting for pip to do its thing. :)
remus 2y ago
Pip doesn't resolve dependencies for you. On small projects that can be ok, but if you're working on something medium to large, or you're working on it with other people you can quickly get yourself into a sticky situation where your environment isn't easily reproducible. Using uv means your project will have well defined dependencies.
- mosselman 2y ago
  Oh wow it doesn’t? What DOES it do then? As I commented here just now I never got pip. This explains it.
  - deleted 2y ago
    [deleted]
  - zo1 2y ago
    The guy doesn't know what he's talking about as pip certainly has dependency resolution. Rather get your python or tech info from a non-flame-war infested thread full of anti-pip and anti-python folk.
- mianos 2y ago
  https://pip.pypa.io/en/stable/topics/dependency-resolution/ https://pip.pypa.io/en/stable/topics/dependency-resolution/
  - remus 2y ago
    My bad, see PaulHoule's comment for what I was getting at.
markerz 2y ago
Yeah, it unifies the whole env experience with the package installation experience. No more forgetting to activate virtualenv first. No more pip installing into the wrong virtual env or accidentally borrowing from the system packages. It’s way easier to specify which version of python to use. Everything is version controlled including python version and variant like cpython, puppy, etc. it’s also REALLY REALLY fast.
2wrist 2y ago
Also you can set the python version for that project. It will download whatever version you need and just use it.
mardifoufs 2y ago
Well, for one you can't actually package or add a local requirement (for example , a vendored package) to the usual pip requirements.txt (or with pyproject.toml, or any other standard way) afaik. I saw a discourse reply that cited some sort of possible security issue but that was basically it and that means that the only way to get that functionality is to not use pip. It's really not a lot of major stuff, just a lot of little paper cuts that makes it a lot easier to just use something else once your project gets to a certain size.
- BeefWellington 2y ago
  Sure you can. It's in their example for how to use requirements.txt: https://pip.pypa.io/en/stable/reference/requirements-file-format/ https://pip.pypa.io/en/stable/reference/requirements-file-fo... Maybe there's some concrete example you have in mind though?
  - mardifoufs 2y ago
    I don't think so, though maybe I didn't explain myself correctly. You can link to a relative package wheel I think, but not to a package repo. So if you have a repo, with your main package in ./src, and you vendor or need a package from another subfolder (let's say ./vendored/freetype) , you can't actually do it in a way that won't break the moment you share your package. You can't put ./vendored/freetype in your requirements.txt, it just fails. That means you either need to use pypi or do an extremely messy hack that involves adding the vendored package as a sub package to your main source, and then do some importlib black magic to make sure that everything uses said package. https://github.com/pypa/pip/issues/6658 https://github.com/pypa/pip/issues/6658 https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/15 https://discuss.python.org/t/what-is-the-correct-interpretat...
    - BeefWellington 2y ago
      In this scenario, reading between the lines, the vendor is not providing a public / published package but does provide the source as like a tarball? I have yet to run into that particular case where the vendor didn't supply their own repo in favour of just providing the source directly. However I do use what are essentially vendor-supplied packages (distant teams in the org) and in those cases I just point at their GitLab/GitHub repo directly. Even for some components within my own team we do it this way.
mosselman 2y ago
I am not a python developer, but sometimes I use python projects. This puts me in a position where I need to get stuff working while knowing almost nothing about how python package management works. Also I don’t recognise errors and I don’t know which python versions generally work well with what. I’ve had it happen so often with pip that I’d have something setup just fine. Let’s say some stable diffusion ui. Then some other month I want to experiment with something like airbyte. Can’t get it working at all. Then some days later I think, let’s generate an image. Only to find out that with pip installing all sorts of stuff for airbyte, I’ve messed up my stable diffusion install somehow. Uv clicked right away for me and I don’t have any of these issues. Was I using pip and asdf incorrectly before? Probably. Was it worth learning how to do it properly in the previous way? Nope. So uv is really great for me.
- hyeonwho4 2y ago
  This is not just a pip problem. I had the problem with anaconda a few years ago where upgrading the built in editor (spyder?) pulled versions of packages which broke my ML code, or made dependencies impossible to reconsile. It was a mess, wasting hours of time. Since then I use one pip venv for each project and just never update dependencies.
  - zahlman 2y ago
    Spyder isn't built-in; IDLE comes with Python (unless you get it via Debian, at least), but is not separately upgradable (as the underlying `idlelib` is part of the standard library). If upgrading Spyder broke your environment, that's presumably because you were using the same environment that Spyder itself was in. (Spyder is also implemented in Python, as the name suggests.) However, IDEs for Python also like to try to do environment management for you (which may conflict with other tools you want to use specifically for the purpose). That's one of the reasons I just edit my code in Vim. If updating dependencies breaks your code, it's ultimately the fault of the dependency (and their maintainers will in turn blame you for not paying attention to their deprecation warnings).
    - matsemann 2y ago
PaulHoule 2y ago
pip's resolving algorithm is not sound. If your Python projects are really simple it seems to work but as your projects get more complex the failure rate creeps up over time. You might pip install something and have it fail and then go back to zero and restart and have it work but at some point that will fail. conda has a correct resolving algorithm but the packages are out of date and add about as many quality problems as they fix. I worked at a place where the engineering manager was absolutely exasperated with the problems we were having with building and deploying AI/ML software in Python. I had figured out pretty much all the problems after about nine months and had developed a 'wheelhouse' procedure for building our system reliably, but it was too late. Not long after I sketched out a system that was a lot like uv but it was written in Python and thus had problems with maintaining its own stable Python enivronment (e.g. poetry seems to trash itself every six months or so.) Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been. (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.) I didn't go forward with developing it because I argued with a lot of people who, like you, thought it was "the perfect being the enemy of the good" when it was really "the incorrect being the enemy of the correct." I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.
- MadnessASAP 2y ago
  May I introduce you to our lord and saviour, Nix and it's most holy child nixpkgs! With only a small tithing of your sanity and ability to Interop with any other dependency management you can free yourself of all dependency woes forever! [] For various broad* definitions of forever. [*] Like, really, really broad** [**] Maybe a week if you're lucky
  - benatkin 2y ago
    The Nix of Python, conda, was already mentioned. > add about as many quality problems as they fix
whimsicalism 2y ago
in my view, depending on your workflow you might have been missing out on pyenv in the past but not really if you feel comfortable self-managing your venvs. now though, yes unequivocally you are missing out.
NeutralCrane 2y ago
Couple of things. - pip doesn't handle your Python executable, just your Python dependencies. So if you want/need to swap between Python versions (3.11 to 3.12 for example), it doesn't give you anything. Generally people use an additional tool such as pyenv to manage this. Tools like uv and Poetry do this as well as handling dependencies - pip doesn't resolve dependencies of dependencies. pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies. I've had many issues where my environment stopped working despite none of my specified dependencies changing, because underlying dependencies introduced breaking changes. To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility. uv and poetry do this out of the box. - Tool usage. Say there is a python package you want to use across many environments without installing in the environments themselves (such as a linting tool like ruff). With pip, you need to install another tool like pipx to install something that can be used across environments. uv can do this out of the box. Plus there is a whole host of jobs that tools like uv and poetry aim to assist with that pip doesn't, namely project creation and management. You can use uv to create a new Python project scaffolding for applications or python modules in a way that conforms with PEP standards with a single command. It also supports workspaces of multiple projects that have separate functionality but require dependencies to be in sync. You can accomplish a lot/all of this using pip with additional tooling, but its a lot more work. And not all use cases will require these.
- kiddico 2y ago
  Sometimes I feel like my up vote doesn't adequately express my gratitude. I appreciate how thorough this was.
- stavros 2y ago
  Oh wow, it actually can handle the Python executable? I didn't know that, that's great! Although it's in the article as well, it didn't click until you said it, thanks!
o11c 2y ago
Much of the Python ecosystem blatantly violates semantic versioning. Most new tooling is designed to work around the bugs introduced by this.
- sgarland 2y ago
  To be fair, Python itself doesn’t follow SemVer. Not in a “they break things they shouldn’t,” but in a “they never claim to be using SemVer.”
- zahlman 2y ago
  Relevant: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/ Semver is hard; you never know what will break at least one of your users (see Hyrum's law), but on the other hand, clear backwards-compatibility breaks will often not affect a large fraction of users - if they preemptively declare that they won't support your next version, they may prevent Pip from finding a set of versions that would actually work just fine.
benreesman 2y ago
Performance and correctness mostly.
coldtea 2y ago
Cool story bro. I've used pip, pyenv, poetry, all are broken in one way or another, and have blind spots they don't serve. If your needs are simple (not mixing Python versions, simple dependencies, not packaging, etc) you can do it with pip, or even with tarballs and make install.
ppierald 2y ago
Respectively, yes. The ability to create venvs so fast, that it becomes a silent operation that the end user never thinks about anymore. The dependency management and installation is lightning quick. It deals with all of the python versioning and I think a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname> Your script code would like: #!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.12" # dependencies = [ # "...", # "..." # ] # /// Then uv will make a new venv, install the dependencies, and execute the script faster than you think. The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling. It is really interesting. You should at least take a look at a YT or something. I think you will be impressed. Good luck!
- zahlman 2y ago
  >Respectively, yes. The ability to create venvs so fast, that it becomes a silent operation that the end user never thinks about anymore. I might just blow your mind here: $ time python -m venv with-pip real 0m3.248s user 0m3.016s sys 0m0.219s $ time python -m venv --without-pip without-pip real 0m0.054s user 0m0.046s sys 0m0.009s The thing that actually takes time is installing Pip into the venv. I already have local demonstrations that this installation can be an order of magnitude faster in native Python. But it's also completely unnecessary to do that: $ source without-pip/bin/activate (without-pip) $ ~/.local/bin/pip --python `which python` install package-installation-test Collecting package-installation-test Using cached package_installation_test-1.0.0-py3-none-any.whl.metadata (3.1 kB) Using cached package_installation_test-1.0.0-py3-none-any.whl (3.1 kB) Installing collected packages: package-installation-test Successfully installed package-installation-test-1.0.0 I have wrappers for this, of course (and I'm explicitly showing the path to a separate Pip that's already on my path for demonstration purposes). > a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname> Yes, Uv implements PEP 723 "Inline Script Metadata" (https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/) - originally the idea of Paul Moore from the Pip dev team, whose competing PEP 722 lost out (see https://discuss.python.org/t/_/29905 https://discuss.python.org/t/_/29905). He's been talking about a feature like this for quite a while, although I can't easily find the older discussion. He seems to consider it out of scope for Pip, but it's also available in Pipx as of version 1.4.2 (https://pipx.pypa.io/stable/CHANGELOG/ https://pipx.pypa.io/stable/CHANGELOG/). > The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling. Part of why Pip is slow at this is because it insists on checking PyPI for newer versions even if it has something cached, and because its internal cache is designed to simulate an Internet connection and go through all the usual metadata parsing etc. instead of just storing the wheels directly. But it's also just slow at actually installing packages when it already has the wheel. In principle, nothing prevents a Python program from doing caching sensibly and from shuffling symlinks around.
rat87 2y ago
Pip is sort of broken before because it encourages confusion between requirements and lock files. In other languages with package managers you generally specify your requirements with ranges and get a lock file with exact versions of those and any transitive dependencies letting you easily recreate a known working environment. The only way to do that in pip is to make a *new* venue install then pip freeze. I think pip tools package is supposed to help but it's a separate tool (one which I've also includes). Also putting stuff in pyproject.toml feels more solid then requirements files (and allows options to be set on requirements (like installing only one package that's only on your company's private python package index mirror while installing the others from the global python package index) and allows dev dependencies and other optional features dependency groups without multiple requirements files and having to update locks on those files. It also automatically creates venvs if you delete them. And it automatically updates packages when you run something with uv run file.py (useful when somebody may have updated the requirements in git). It also lets you install self contained (installed in a virtualenv and linked to ~/.local/bin which is added to your path)python tools (replacing pipx). It installs self contained python builds letting you more easily pick python version and specify it in a .python-version file for your project (replacing pyenv and usually much nicer because pyenv compiles them locally) Uv also makes it easier to explore and say start a ipython shell with 2 libraries uv run --with ipython --with colorful --with https ipython It caches downloads. Of course the http itself isn't faster but they're exploring things to speed that part up and since it's written in rust local stuff (like deleting and recreating a venv with cached packages) tends to be blazing fast
__mharrison__ 2y ago
I was in your boat too. Been using Python since 2000 and pretty satisfied with venv and pip. However, the speed alone is reason enough to switch. Try it once and you will be sold.
cpburns2009 2y ago
I'm fairly minimalist when it comes to tooling: venv, pip, and pip-tools. I've started to use uv recently because it resolves packages significantly faster than pip/pip-tools. It will generate a "requirements.txt" with 30 packages in a few seconds rather than a minute or two.
ayjay_t 2y ago
Yeah, I switched from pip to uv. uv seems like its almost the perfect solution for me. it does virtualenv, it does pyenv, it does pip, so all thats managed in once place. its much faster than pip. its like 80% of my workflow now.

mplewis 2y ago

Pip only has requirements.txt and doesn't have lockfiles, so you can't guarantee that the bugs you're seeing on your system are the same as the bugs on your production system.

aidos 2y ago
I’ve always worked around that by having a requirements.base.txt and a requirements.txt for the locked versions. Obviously pip doesn’t do that for you but it’s not hard to manage yourself. Having said that, I’m going to give uv a shot because I hear so many good things about it.
- mikepurvis 2y ago
  I’m grouchy because I finally got religion on poetry a few years ago, but the hype on uv is good enough that I’ll have to give it a shot.
  - kstrauser 2y ago
    I freaking love Poetry. It was a huge breath of fresh air after years of pip and a short detour with Pipenv. If uv stopped existing I’d go back to Poetry. But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.
  - remram 2y ago
    With the new major release of Poetry that just came out I also feel like it might be a good time to switch to Uv rather than adapt to this new version: https://python-poetry.org/blog/announcing-poetry-2.0.0/ https://python-poetry.org/blog/announcing-poetry-2.0.0/
- selcuka 2y ago
  With pip the best practice is to have a requirements.txt with direct requirements (strictly or loosely pinned), and a separate constraints.txt file [1] with strictly pinned versions of all direct- and sub-dependencies (basically the output of `pip freeze`). The latter works like a lock file. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
  - zahlman 2y ago
    For direct requirements you're better off using the `pyproject.toml` for direct dependencies (and you can plausibly use it to pin everything if you're developing an application). It's project metadata that you'll need anyway for building your project, and the "editable wheel" hack allows Pip to use that information to set up an environment for your code (via `pip install -e .`).
- remram 2y ago
  This works until you need to upgrade something, pip might upgrade to a broken set of dependencies. Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.
  - zahlman 2y ago
    > pip might upgrade to a broken set of dependencies. I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end. > Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix. The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo: $ pip install 'numpy;python_version<="2.7"' Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment > There are a lot of gotchas that pip can't fix. There are a lot of serious problems with Pip - I just don't think these are among them.
    - remram 2y ago
      You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.
      - zahlman 2y ago
        Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)
zo1 2y ago
The requirements.txt file is the lockfile. Anyways, this whole obsession with locked deps or "lockfiles" is such an anti-pattern, I have no idea why we went there as an industry. Probably as a result of some of the newer stuff that is classified as "hipster-tech" such as docker and javascript.
- n144q 2y ago
  Just because you don't understand it, it's ok to call it an "anti-pattern"? Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you. If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.
  - driggs 2y ago
    Just because someone has a different perspective than you doesn't mean they don't "understand". Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.
    - Uvix 2y ago
      If you're developing a library, and you have a requirement for what's normally a transitive dependency, it should be specified as a top-level dependency.
      - driggs 2y ago
        The point is that if I'm writing a library and I specify `requests == 1.2.3`, then what are you going to do in your application if you need both my library and `requests == 1.2.4`? This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible. It's the developers of an application who should use a lockfile to lock transitive dependencies.
polski-g 2y ago
"pip freeze" generates a lockfile.
- orf 2y ago
  No, that generates a list of currently installed packages. That’s very much not a lock file, even if it is possible to abuse it as such.
  - deleted 2y ago
    [deleted]
  - zahlman 2y ago
    A list of currently installed packages in the current environment, with their exact versions. This is only the actually needed packages, with their transitive dependencies, unless you've left something behind from earlier in development. If you're keeping abstract dependencies up to date in `pyproject.toml` (which you need to do anyway to build and release the project), you can straightforwardly re-create the environment from that list and freeze whatever solution you get (after testing).
    - orf 2y ago
      Doesn’t account for differences in platforms or Python versions, and doesn’t contain resolved dependency hashes. So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.
      - zahlman 2y ago
        Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution. >only works on the exact same Python version It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each. In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-a-tomlitomllib-compatibility-layer https://github.com/hukkin/tomli?tab=readme-ov-file#building-...). >on my machine (Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.) If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.) This is also only relevant for projects that include non-Python code that requires a build step, of course. >assuming no dependencies have published new packages. PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.

amluto 2y ago

If you switch to uv, you’ll have fewer excuses to take coffee breaks while waiting for pip to do its thing. :)

remus 2y ago

Pip doesn't resolve dependencies for you. On small projects that can be ok, but if you're working on something medium to large, or you're working on it with other people you can quickly get yourself into a sticky situation where your environment isn't easily reproducible. Using uv means your project will have well defined dependencies.

mosselman 2y ago
Oh wow it doesn’t? What DOES it do then? As I commented here just now I never got pip. This explains it.
- deleted 2y ago
  [deleted]
- zo1 2y ago
  The guy doesn't know what he's talking about as pip certainly has dependency resolution. Rather get your python or tech info from a non-flame-war infested thread full of anti-pip and anti-python folk.
mianos 2y ago
https://pip.pypa.io/en/stable/topics/dependency-resolution/ https://pip.pypa.io/en/stable/topics/dependency-resolution/
- remus 2y ago
  My bad, see PaulHoule's comment for what I was getting at.

markerz 2y ago

Yeah, it unifies the whole env experience with the package installation experience. No more forgetting to activate virtualenv first. No more pip installing into the wrong virtual env or accidentally borrowing from the system packages. It’s way easier to specify which version of python to use. Everything is version controlled including python version and variant like cpython, puppy, etc. it’s also REALLY REALLY fast.

2wrist 2y ago

Also you can set the python version for that project. It will download whatever version you need and just use it.

mardifoufs 2y ago

Well, for one you can't actually package or add a local requirement (for example , a vendored package) to the usual pip requirements.txt (or with pyproject.toml, or any other standard way) afaik. I saw a discourse reply that cited some sort of possible security issue but that was basically it and that means that the only way to get that functionality is to not use pip. It's really not a lot of major stuff, just a lot of little paper cuts that makes it a lot easier to just use something else once your project gets to a certain size.

BeefWellington 2y ago
Sure you can. It's in their example for how to use requirements.txt: https://pip.pypa.io/en/stable/reference/requirements-file-format/ https://pip.pypa.io/en/stable/reference/requirements-file-fo... Maybe there's some concrete example you have in mind though?
- mardifoufs 2y ago
  I don't think so, though maybe I didn't explain myself correctly. You can link to a relative package wheel I think, but not to a package repo. So if you have a repo, with your main package in ./src, and you vendor or need a package from another subfolder (let's say ./vendored/freetype) , you can't actually do it in a way that won't break the moment you share your package. You can't put ./vendored/freetype in your requirements.txt, it just fails. That means you either need to use pypi or do an extremely messy hack that involves adding the vendored package as a sub package to your main source, and then do some importlib black magic to make sure that everything uses said package. https://github.com/pypa/pip/issues/6658 https://github.com/pypa/pip/issues/6658 https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/15 https://discuss.python.org/t/what-is-the-correct-interpretat...
  - BeefWellington 2y ago
    In this scenario, reading between the lines, the vendor is not providing a public / published package but does provide the source as like a tarball? I have yet to run into that particular case where the vendor didn't supply their own repo in favour of just providing the source directly. However I do use what are essentially vendor-supplied packages (distant teams in the org) and in those cases I just point at their GitLab/GitHub repo directly. Even for some components within my own team we do it this way.

mosselman 2y ago

I am not a python developer, but sometimes I use python projects. This puts me in a position where I need to get stuff working while knowing almost nothing about how python package management works. Also I don’t recognise errors and I don’t know which python versions generally work well with what. I’ve had it happen so often with pip that I’d have something setup just fine. Let’s say some stable diffusion ui. Then some other month I want to experiment with something like airbyte. Can’t get it working at all. Then some days later I think, let’s generate an image. Only to find out that with pip installing all sorts of stuff for airbyte, I’ve messed up my stable diffusion install somehow. Uv clicked right away for me and I don’t have any of these issues. Was I using pip and asdf incorrectly before? Probably. Was it worth learning how to do it properly in the previous way? Nope. So uv is really great for me.

hyeonwho4 2y ago
This is not just a pip problem. I had the problem with anaconda a few years ago where upgrading the built in editor (spyder?) pulled versions of packages which broke my ML code, or made dependencies impossible to reconsile. It was a mess, wasting hours of time. Since then I use one pip venv for each project and just never update dependencies.
- zahlman 2y ago
  Spyder isn't built-in; IDLE comes with Python (unless you get it via Debian, at least), but is not separately upgradable (as the underlying `idlelib` is part of the standard library). If upgrading Spyder broke your environment, that's presumably because you were using the same environment that Spyder itself was in. (Spyder is also implemented in Python, as the name suggests.) However, IDEs for Python also like to try to do environment management for you (which may conflict with other tools you want to use specifically for the purpose). That's one of the reasons I just edit my code in Vim. If updating dependencies breaks your code, it's ultimately the fault of the dependency (and their maintainers will in turn blame you for not paying attention to their deprecation warnings).
  - matsemann 2y ago

PaulHoule 2y ago

pip's resolving algorithm is not sound. If your Python projects are really simple it seems to work but as your projects get more complex the failure rate creeps up over time. You might pip install something and have it fail and then go back to zero and restart and have it work but at some point that will fail. conda has a correct resolving algorithm but the packages are out of date and add about as many quality problems as they fix. I worked at a place where the engineering manager was absolutely exasperated with the problems we were having with building and deploying AI/ML software in Python. I had figured out pretty much all the problems after about nine months and had developed a 'wheelhouse' procedure for building our system reliably, but it was too late. Not long after I sketched out a system that was a lot like uv but it was written in Python and thus had problems with maintaining its own stable Python enivronment (e.g. poetry seems to trash itself every six months or so.) Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been. (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.) I didn't go forward with developing it because I argued with a lot of people who, like you, thought it was "the perfect being the enemy of the good" when it was really "the incorrect being the enemy of the correct." I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.

MadnessASAP 2y ago
May I introduce you to our lord and saviour, Nix and it's most holy child nixpkgs! With only a small tithing of your sanity and ability to Interop with any other dependency management you can free yourself of all dependency woes forever! [] For various broad* definitions of forever. [*] Like, really, really broad** [**] Maybe a week if you're lucky
- benatkin 2y ago
  The Nix of Python, conda, was already mentioned. > add about as many quality problems as they fix

whimsicalism 2y ago

in my view, depending on your workflow you might have been missing out on pyenv in the past but not really if you feel comfortable self-managing your venvs. now though, yes unequivocally you are missing out.

NeutralCrane 2y ago

Couple of things. - pip doesn't handle your Python executable, just your Python dependencies. So if you want/need to swap between Python versions (3.11 to 3.12 for example), it doesn't give you anything. Generally people use an additional tool such as pyenv to manage this. Tools like uv and Poetry do this as well as handling dependencies - pip doesn't resolve dependencies of dependencies. pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies. I've had many issues where my environment stopped working despite none of my specified dependencies changing, because underlying dependencies introduced breaking changes. To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility. uv and poetry do this out of the box. - Tool usage. Say there is a python package you want to use across many environments without installing in the environments themselves (such as a linting tool like ruff). With pip, you need to install another tool like pipx to install something that can be used across environments. uv can do this out of the box. Plus there is a whole host of jobs that tools like uv and poetry aim to assist with that pip doesn't, namely project creation and management. You can use uv to create a new Python project scaffolding for applications or python modules in a way that conforms with PEP standards with a single command. It also supports workspaces of multiple projects that have separate functionality but require dependencies to be in sync. You can accomplish a lot/all of this using pip with additional tooling, but its a lot more work. And not all use cases will require these.

kiddico 2y ago
Sometimes I feel like my up vote doesn't adequately express my gratitude. I appreciate how thorough this was.
stavros 2y ago
Oh wow, it actually can handle the Python executable? I didn't know that, that's great! Although it's in the article as well, it didn't click until you said it, thanks!

o11c 2y ago

Much of the Python ecosystem blatantly violates semantic versioning. Most new tooling is designed to work around the bugs introduced by this.

sgarland 2y ago
To be fair, Python itself doesn’t follow SemVer. Not in a “they break things they shouldn’t,” but in a “they never claim to be using SemVer.”
zahlman 2y ago
Relevant: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/ Semver is hard; you never know what will break at least one of your users (see Hyrum's law), but on the other hand, clear backwards-compatibility breaks will often not affect a large fraction of users - if they preemptively declare that they won't support your next version, they may prevent Pip from finding a set of versions that would actually work just fine.

benreesman 2y ago

Performance and correctness mostly.

coldtea 2y ago

Cool story bro. I've used pip, pyenv, poetry, all are broken in one way or another, and have blind spots they don't serve. If your needs are simple (not mixing Python versions, simple dependencies, not packaging, etc) you can do it with pip, or even with tarballs and make install.

ppierald 2y ago

Respectively, yes. The ability to create venvs so fast, that it becomes a silent operation that the end user never thinks about anymore. The dependency management and installation is lightning quick. It deals with all of the python versioning and I think a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname> Your script code would like: #!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.12" # dependencies = [ # "...", # "..." # ] # /// Then uv will make a new venv, install the dependencies, and execute the script faster than you think. The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling. It is really interesting. You should at least take a look at a YT or something. I think you will be impressed. Good luck!

zahlman 2y ago
>Respectively, yes. The ability to create venvs so fast, that it becomes a silent operation that the end user never thinks about anymore. I might just blow your mind here: $ time python -m venv with-pip real 0m3.248s user 0m3.016s sys 0m0.219s $ time python -m venv --without-pip without-pip real 0m0.054s user 0m0.046s sys 0m0.009s The thing that actually takes time is installing Pip into the venv. I already have local demonstrations that this installation can be an order of magnitude faster in native Python. But it's also completely unnecessary to do that: $ source without-pip/bin/activate (without-pip) $ ~/.local/bin/pip --python `which python` install package-installation-test Collecting package-installation-test Using cached package_installation_test-1.0.0-py3-none-any.whl.metadata (3.1 kB) Using cached package_installation_test-1.0.0-py3-none-any.whl (3.1 kB) Installing collected packages: package-installation-test Successfully installed package-installation-test-1.0.0 I have wrappers for this, of course (and I'm explicitly showing the path to a separate Pip that's already on my path for demonstration purposes). > a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname> Yes, Uv implements PEP 723 "Inline Script Metadata" (https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/) - originally the idea of Paul Moore from the Pip dev team, whose competing PEP 722 lost out (see https://discuss.python.org/t/_/29905 https://discuss.python.org/t/_/29905). He's been talking about a feature like this for quite a while, although I can't easily find the older discussion. He seems to consider it out of scope for Pip, but it's also available in Pipx as of version 1.4.2 (https://pipx.pypa.io/stable/CHANGELOG/ https://pipx.pypa.io/stable/CHANGELOG/). > The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling. Part of why Pip is slow at this is because it insists on checking PyPI for newer versions even if it has something cached, and because its internal cache is designed to simulate an Internet connection and go through all the usual metadata parsing etc. instead of just storing the wheels directly. But it's also just slow at actually installing packages when it already has the wheel. In principle, nothing prevents a Python program from doing caching sensibly and from shuffling symlinks around.

rat87 2y ago

Pip is sort of broken before because it encourages confusion between requirements and lock files. In other languages with package managers you generally specify your requirements with ranges and get a lock file with exact versions of those and any transitive dependencies letting you easily recreate a known working environment. The only way to do that in pip is to make a *new* venue install then pip freeze. I think pip tools package is supposed to help but it's a separate tool (one which I've also includes). Also putting stuff in pyproject.toml feels more solid then requirements files (and allows options to be set on requirements (like installing only one package that's only on your company's private python package index mirror while installing the others from the global python package index) and allows dev dependencies and other optional features dependency groups without multiple requirements files and having to update locks on those files. It also automatically creates venvs if you delete them. And it automatically updates packages when you run something with uv run file.py (useful when somebody may have updated the requirements in git). It also lets you install self contained (installed in a virtualenv and linked to ~/.local/bin which is added to your path)python tools (replacing pipx). It installs self contained python builds letting you more easily pick python version and specify it in a .python-version file for your project (replacing pyenv and usually much nicer because pyenv compiles them locally) Uv also makes it easier to explore and say start a ipython shell with 2 libraries uv run --with ipython --with colorful --with https ipython It caches downloads. Of course the http itself isn't faster but they're exploring things to speed that part up and since it's written in rust local stuff (like deleting and recreating a venv with cached packages) tends to be blazing fast

__mharrison__ 2y ago

I was in your boat too. Been using Python since 2000 and pretty satisfied with venv and pip. However, the speed alone is reason enough to switch. Try it once and you will be sold.

cpburns2009 2y ago

I'm fairly minimalist when it comes to tooling: venv, pip, and pip-tools. I've started to use uv recently because it resolves packages significantly faster than pip/pip-tools. It will generate a "requirements.txt" with 30 packages in a few seconds rather than a minute or two.

ayjay_t 2y ago

Yeah, I switched from pip to uv. uv seems like its almost the perfect solution for me. it does virtualenv, it does pyenv, it does pip, so all thats managed in once place. its much faster than pip. its like 80% of my workflow now.