4 ms·

Python is unified on a requirements.txt file with pip, to manage dependencies.

woodruffw 6y ago
I don’t think this is true. I’ve personally seen the following (and I’m positive there are others): 1. setup.py with self contained dependencies (this is my general preference) 2. setup.py loading from requirements.txt 3. Standalone requirements.txt 4. requirements.txt generated from requirements.in (or similar) 5. pyproject.toml + Poetry with a dependencies section
wwright 6y ago
requirements.txt has a lot of deficiencies for package management (compare to the various features Yarn or Cargo offer). The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies, and even then there’s no verification that the content matches what is expected beyond the version number (other tools use a content checksum). This can cause a long tail of reproducibility and maintenance issues. It’s also a tiny subset of what the parent describes.
- bpicolo 6y ago
  It also can't separate dev vs test vs prod dependencies, for example. Maintaining separate files for those is not exactly a convenient solution
- Sir_Substance 6y ago
  When I first set up my requirements.txt's, I usually set up a venv, install the things that I need and then run "pip freeze" to get a list of all deps including transitive deps, and put them all in the requirements.txts. I do sometimes feel that people are making rube goldberg machines out of their package management in an attempt to avoid just writing down all their deps.
- gruez 6y ago
  >The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies Isn't that what pip freeze is for?
  - lmm 6y ago
    The trouble there is that if you freeze you can't unfreeze. A project needs the equivalent of both gemfile and gemfile.lock (unless you're using a system like maven that got things right the first time), but there's no standard way of having both of them in Python.
- HALtheWise 6y ago
  I strongly recommend https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to solve this. It provides a simple script to take a requirements file and "compile" a full specification of all transitive dependencies. You check both files into the repo, point pip at the generated file, and manually modify the other one. It means you often don't need to pin requirements manually at all, and the versions will be explicitly updated whenever you choose to recompile your requirements.
jniedrauer 6y ago
This is a big oversimplification. There are often multiple setup.txt files for different environments, virtualenv setup scripts, tox files, several different build config files, manifest files, and so on. Python probably has more magic config files than C#, which is quite an accomplishment.
klyrs 6y ago
Not "a" requirements.txt file, sadly. I've seen packages with different requirements for building, installing, and testing. All with their own damned txt file. Oh! And don't forget the duplication in setup.py -- you'll want to put your requirements there, too.
- 1f60c 6y ago
  As far as I know, Pip doesn't have the notion of dev dependencies, so you need at least two requirements[.dev].txt files.
  - dragonwriter 6y ago
    > As far as I know, Python doesn't have the notion of dev dependencies Python doesn't have the notion of dependencies. Pip doesn't have dev dependencies, poetry does.
    - 1f60c 6y ago
      D’oh! Thanks. :)
    - pas 6y ago
      That's (the deps, not the dev deps) likely to change with pip 20.3! All hail the dependency resolver. It's already available behind a feature flag. https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-changes-20-3.html https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-...
  - deleted 6y ago
    [deleted]
jjoonathan 6y ago
But there are usually several versions of python with several pips, possibly in combination with anaconda with multiple environments, a combination of named and rolling releases, and a SAT solver that's always slow but occasionally spontaneously decides to become extra slow and extend your build time by hours. Oh and I forgot about setup.py. That too.
dfee 6y ago
I tried to downvote this, but I think it’s already downvoted to the max. There is so much wrong with this pattern. Please use something like Poetry to manage deps.
paulie_a 6y ago
I never have a unified requirements file.

I don’t think this is true. I’ve personally seen the following (and I’m positive there are others): 1. setup.py with self contained dependencies (this is my general preference) 2. setup.py loading from requirements.txt 3. Standalone requirements.txt 4. requirements.txt generated from requirements.in (or similar) 5. pyproject.toml + Poetry with a dependencies section

wwright 6y ago

requirements.txt has a lot of deficiencies for package management (compare to the various features Yarn or Cargo offer). The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies, and even then there’s no verification that the content matches what is expected beyond the version number (other tools use a content checksum). This can cause a long tail of reproducibility and maintenance issues. It’s also a tiny subset of what the parent describes.

bpicolo 6y ago
It also can't separate dev vs test vs prod dependencies, for example. Maintaining separate files for those is not exactly a convenient solution
Sir_Substance 6y ago
When I first set up my requirements.txt's, I usually set up a venv, install the things that I need and then run "pip freeze" to get a list of all deps including transitive deps, and put them all in the requirements.txts. I do sometimes feel that people are making rube goldberg machines out of their package management in an attempt to avoid just writing down all their deps.
gruez 6y ago
>The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies Isn't that what pip freeze is for?
- lmm 6y ago
  The trouble there is that if you freeze you can't unfreeze. A project needs the equivalent of both gemfile and gemfile.lock (unless you're using a system like maven that got things right the first time), but there's no standard way of having both of them in Python.
HALtheWise 6y ago
I strongly recommend https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to solve this. It provides a simple script to take a requirements file and "compile" a full specification of all transitive dependencies. You check both files into the repo, point pip at the generated file, and manually modify the other one. It means you often don't need to pin requirements manually at all, and the versions will be explicitly updated whenever you choose to recompile your requirements.

jniedrauer 6y ago

This is a big oversimplification. There are often multiple setup.txt files for different environments, virtualenv setup scripts, tox files, several different build config files, manifest files, and so on. Python probably has more magic config files than C#, which is quite an accomplishment.

klyrs 6y ago

Not "a" requirements.txt file, sadly. I've seen packages with different requirements for building, installing, and testing. All with their own damned txt file. Oh! And don't forget the duplication in setup.py -- you'll want to put your requirements there, too.

1f60c 6y ago
As far as I know, Pip doesn't have the notion of dev dependencies, so you need at least two requirements[.dev].txt files.
- dragonwriter 6y ago
  > As far as I know, Python doesn't have the notion of dev dependencies Python doesn't have the notion of dependencies. Pip doesn't have dev dependencies, poetry does.
  - 1f60c 6y ago
    D’oh! Thanks. :)
  - pas 6y ago
    That's (the deps, not the dev deps) likely to change with pip 20.3! All hail the dependency resolver. It's already available behind a feature flag. https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-changes-20-3.html https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-...
- deleted 6y ago
  [deleted]

jjoonathan 6y ago

But there are usually several versions of python with several pips, possibly in combination with anaconda with multiple environments, a combination of named and rolling releases, and a SAT solver that's always slow but occasionally spontaneously decides to become extra slow and extend your build time by hours. Oh and I forgot about setup.py. That too.

dfee 6y ago

I tried to downvote this, but I think it’s already downvoted to the max. There is so much wrong with this pattern. Please use something like Poetry to manage deps.

paulie_a 6y ago

I never have a unified requirements file.