6 ms·

Pip only has requirements.txt and doesn't have lockfiles, so you can't guarantee that the bugs you're seeing on your system are the same as the bugs on your pro

Pip only has requirements.txt and doesn't have lockfiles, so you can't guarantee that the bugs you're seeing on your system are the same as the bugs on your production system.

aidos 2y ago
I’ve always worked around that by having a requirements.base.txt and a requirements.txt for the locked versions. Obviously pip doesn’t do that for you but it’s not hard to manage yourself. Having said that, I’m going to give uv a shot because I hear so many good things about it.
- mikepurvis 2y ago
  I’m grouchy because I finally got religion on poetry a few years ago, but the hype on uv is good enough that I’ll have to give it a shot.
  - kstrauser 2y ago
    I freaking love Poetry. It was a huge breath of fresh air after years of pip and a short detour with Pipenv. If uv stopped existing I’d go back to Poetry. But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.
  - remram 2y ago
    With the new major release of Poetry that just came out I also feel like it might be a good time to switch to Uv rather than adapt to this new version: https://python-poetry.org/blog/announcing-poetry-2.0.0/ https://python-poetry.org/blog/announcing-poetry-2.0.0/
- selcuka 2y ago
  With pip the best practice is to have a requirements.txt with direct requirements (strictly or loosely pinned), and a separate constraints.txt file [1] with strictly pinned versions of all direct- and sub-dependencies (basically the output of `pip freeze`). The latter works like a lock file. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
  - zahlman 2y ago
    For direct requirements you're better off using the `pyproject.toml` for direct dependencies (and you can plausibly use it to pin everything if you're developing an application). It's project metadata that you'll need anyway for building your project, and the "editable wheel" hack allows Pip to use that information to set up an environment for your code (via `pip install -e .`).
- remram 2y ago
  This works until you need to upgrade something, pip might upgrade to a broken set of dependencies. Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.
  - zahlman 2y ago
    > pip might upgrade to a broken set of dependencies. I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end. > Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix. The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo: $ pip install 'numpy;python_version<="2.7"' Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment > There are a lot of gotchas that pip can't fix. There are a lot of serious problems with Pip - I just don't think these are among them.
    - remram 2y ago
      You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.
      - zahlman 2y ago
        Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)
zo1 2y ago
The requirements.txt file is the lockfile. Anyways, this whole obsession with locked deps or "lockfiles" is such an anti-pattern, I have no idea why we went there as an industry. Probably as a result of some of the newer stuff that is classified as "hipster-tech" such as docker and javascript.
- n144q 2y ago
  Just because you don't understand it, it's ok to call it an "anti-pattern"? Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you. If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.
  - driggs 2y ago
    Just because someone has a different perspective than you doesn't mean they don't "understand". Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.
    - Uvix 2y ago
      If you're developing a library, and you have a requirement for what's normally a transitive dependency, it should be specified as a top-level dependency.
      - driggs 2y ago
        The point is that if I'm writing a library and I specify `requests == 1.2.3`, then what are you going to do in your application if you need both my library and `requests == 1.2.4`? This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible. It's the developers of an application who should use a lockfile to lock transitive dependencies.
polski-g 2y ago
"pip freeze" generates a lockfile.
- orf 2y ago
  No, that generates a list of currently installed packages. That’s very much not a lock file, even if it is possible to abuse it as such.
  - deleted 2y ago
    [deleted]
  - zahlman 2y ago
    A list of currently installed packages in the current environment, with their exact versions. This is only the actually needed packages, with their transitive dependencies, unless you've left something behind from earlier in development. If you're keeping abstract dependencies up to date in `pyproject.toml` (which you need to do anyway to build and release the project), you can straightforwardly re-create the environment from that list and freeze whatever solution you get (after testing).
    - orf 2y ago
      Doesn’t account for differences in platforms or Python versions, and doesn’t contain resolved dependency hashes. So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.
      - zahlman 2y ago
        Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution. >only works on the exact same Python version It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each. In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-a-tomlitomllib-compatibility-layer https://github.com/hukkin/tomli?tab=readme-ov-file#building-...). >on my machine (Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.) If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.) This is also only relevant for projects that include non-Python code that requires a build step, of course. >assuming no dependencies have published new packages. PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.

Pip only has requirements.txt and doesn't have lockfiles, so you can't guarantee that the bugs you're seeing on your system are the same as the bugs on your production system.

aidos 2y ago

I’ve always worked around that by having a requirements.base.txt and a requirements.txt for the locked versions. Obviously pip doesn’t do that for you but it’s not hard to manage yourself. Having said that, I’m going to give uv a shot because I hear so many good things about it.

mikepurvis 2y ago
I’m grouchy because I finally got religion on poetry a few years ago, but the hype on uv is good enough that I’ll have to give it a shot.
- kstrauser 2y ago
  I freaking love Poetry. It was a huge breath of fresh air after years of pip and a short detour with Pipenv. If uv stopped existing I’d go back to Poetry. But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.
- remram 2y ago
  With the new major release of Poetry that just came out I also feel like it might be a good time to switch to Uv rather than adapt to this new version: https://python-poetry.org/blog/announcing-poetry-2.0.0/ https://python-poetry.org/blog/announcing-poetry-2.0.0/
selcuka 2y ago
With pip the best practice is to have a requirements.txt with direct requirements (strictly or loosely pinned), and a separate constraints.txt file [1] with strictly pinned versions of all direct- and sub-dependencies (basically the output of `pip freeze`). The latter works like a lock file. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
- zahlman 2y ago
  For direct requirements you're better off using the `pyproject.toml` for direct dependencies (and you can plausibly use it to pin everything if you're developing an application). It's project metadata that you'll need anyway for building your project, and the "editable wheel" hack allows Pip to use that information to set up an environment for your code (via `pip install -e .`).
remram 2y ago
This works until you need to upgrade something, pip might upgrade to a broken set of dependencies. Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.
- zahlman 2y ago
  > pip might upgrade to a broken set of dependencies. I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end. > Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix. The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo: $ pip install 'numpy;python_version<="2.7"' Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment > There are a lot of gotchas that pip can't fix. There are a lot of serious problems with Pip - I just don't think these are among them.
  - remram 2y ago
    You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.
    - zahlman 2y ago
      Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)

zo1 2y ago

The requirements.txt file is the lockfile. Anyways, this whole obsession with locked deps or "lockfiles" is such an anti-pattern, I have no idea why we went there as an industry. Probably as a result of some of the newer stuff that is classified as "hipster-tech" such as docker and javascript.

n144q 2y ago
Just because you don't understand it, it's ok to call it an "anti-pattern"? Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you. If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.
- driggs 2y ago
  Just because someone has a different perspective than you doesn't mean they don't "understand". Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.
  - Uvix 2y ago
    If you're developing a library, and you have a requirement for what's normally a transitive dependency, it should be specified as a top-level dependency.
    - driggs 2y ago
      The point is that if I'm writing a library and I specify `requests == 1.2.3`, then what are you going to do in your application if you need both my library and `requests == 1.2.4`? This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible. It's the developers of an application who should use a lockfile to lock transitive dependencies.

polski-g 2y ago

"pip freeze" generates a lockfile.

orf 2y ago
No, that generates a list of currently installed packages. That’s very much not a lock file, even if it is possible to abuse it as such.
- deleted 2y ago
  [deleted]
- zahlman 2y ago
  A list of currently installed packages in the current environment, with their exact versions. This is only the actually needed packages, with their transitive dependencies, unless you've left something behind from earlier in development. If you're keeping abstract dependencies up to date in `pyproject.toml` (which you need to do anyway to build and release the project), you can straightforwardly re-create the environment from that list and freeze whatever solution you get (after testing).
  - orf 2y ago
    Doesn’t account for differences in platforms or Python versions, and doesn’t contain resolved dependency hashes. So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.
    - zahlman 2y ago
      Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution. >only works on the exact same Python version It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each. In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-a-tomlitomllib-compatibility-layer https://github.com/hukkin/tomli?tab=readme-ov-file#building-...). >on my machine (Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.) If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.) This is also only relevant for projects that include non-Python code that requires a build step, of course. >assuming no dependencies have published new packages. PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.