4 ms·

What does this tool provide over an in-project .venv/ directory created by venv and populated by pip from an etc/pip/requirement.txt file containing all the pac

by gjvc 6y ago

What does this tool provide over an in-project .venv/ directory created by venv and populated by pip from an etc/pip/requirement.txt file containing all the packages by version? There seems to be a lot of activity in this area for a seemingly solved problem. Having solved it, people appear to wish to expand the problem to something bigger (such as multiple interpreter versions, for example) I believe the most-common case to be: 1 Choose python version and install on operating system., 2 create .venv/ directory. 3, install packages from package system (pip)., 4 freeze versions to local file. 5 use installed .venv/ and do work. I've got some scripts I use for the above [1] -- must give it some TLC, and I don't understand why people furiously want more than this. [1] https://github.com/gjvc/python-template-project https://github.com/gjvc/python-template-project

Macha 6y ago
The consensus among non-Python developers is that that is not good enough. The problems are: requirements.txt is way too flexible, so users want a lock file to freeze stuff in place. You can use pip freeze to generate a sort-of-lockfile but it's more manual than tools like Cargo/npm/bundler which do it automatically. And more adhoc as to what you call it. Combined with needing to mange venvs and sourcing etc. people want a standard script that just figures out its context from the now standard "explicit requirements"/lockfile setup. It doesn't handle versioning Python itself, which is expected of it for some reason, even though nobody cares that npm/Cargo/bundler don't version their languages.
- divbzero 6y ago
  The lock file shortcoming is better remedied by pip-tools [1] as recommended by a neighboring comment. [1]: https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
- shadycuz 6y ago
  I felt the same way about only needing requirements.txt until I managed a project that needed to be compatible across many versions of python and some of it's dependencies were renamed from one version to another. I highly recommend you take a look at the hypermodern python guide https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
- mumblemumble 6y ago
  Pipenv and poetry help, but, even with them, it's still not good enough. setup.py is a real challenge. Packages being able to execute arbitrary code at install time can make it very difficult to get a truly reproducible python build. Details of how pip (which pipenv and poetry still use) handles packages means that doing anything that even hints of a monorepo is a delicate subject on the best of days.
divbzero 6y ago
Agreed. And the solution to the multiple interpreter versions problem should be orthogonal to venv like pyenv [1] instead of overlapping with venv like pipenv. [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv
mbyio 6y ago
It is a solved problem for other languages. Pip has soooo many issues. The biggest IMO is pip doesn't provide a way to pin indirect dependencies, so there is no good way to ensure that other people working on a project are also using the same dependency versions. You can do it with extra tooling of course - and that's what poetry and pipenv do.
- ak217 6y ago
  Isn’t this what pip freeze is for?
  - Barrin92 6y ago
    pip freeze only pins the version of direct, not transitive dependencies. Meaning you don't get deterministic builds.
    - CrLf 6y ago
      What do you mean? If you run pip freeze inside a venv it will print every user-installed module and their dependencies. When installing them in a new venv as a requirements file, you end up with the exact same set of modules.
      - gjvc 6y ago
        Agree. Works fine for me exactly as you describe. I don't understand why people furiously want more than this for the common case. (Choose python version, create .venv/ directory, install packages, use installed .venv/ and do work.)
        lazypenguin 6y ago
        What happens when you use a different version of the same library that a dependency uses?
        6y ago
gjvc 6y ago
aka UNIX maxim of tools which do one (and only one) thing, and do it well

by gjvc 6y ago

Macha 6y ago

The consensus among non-Python developers is that that is not good enough. The problems are: requirements.txt is way too flexible, so users want a lock file to freeze stuff in place. You can use pip freeze to generate a sort-of-lockfile but it's more manual than tools like Cargo/npm/bundler which do it automatically. And more adhoc as to what you call it. Combined with needing to mange venvs and sourcing etc. people want a standard script that just figures out its context from the now standard "explicit requirements"/lockfile setup. It doesn't handle versioning Python itself, which is expected of it for some reason, even though nobody cares that npm/Cargo/bundler don't version their languages.

divbzero 6y ago
The lock file shortcoming is better remedied by pip-tools [1] as recommended by a neighboring comment. [1]: https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
shadycuz 6y ago
I felt the same way about only needing requirements.txt until I managed a project that needed to be compatible across many versions of python and some of it's dependencies were renamed from one version to another. I highly recommend you take a look at the hypermodern python guide https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
mumblemumble 6y ago
Pipenv and poetry help, but, even with them, it's still not good enough. setup.py is a real challenge. Packages being able to execute arbitrary code at install time can make it very difficult to get a truly reproducible python build. Details of how pip (which pipenv and poetry still use) handles packages means that doing anything that even hints of a monorepo is a delicate subject on the best of days.

divbzero 6y ago

Agreed. And the solution to the multiple interpreter versions problem should be orthogonal to venv like pyenv [1] instead of overlapping with venv like pipenv. [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv

mbyio 6y ago

It is a solved problem for other languages. Pip has soooo many issues. The biggest IMO is pip doesn't provide a way to pin indirect dependencies, so there is no good way to ensure that other people working on a project are also using the same dependency versions. You can do it with extra tooling of course - and that's what poetry and pipenv do.

ak217 6y ago
Isn’t this what pip freeze is for?
- Barrin92 6y ago
  pip freeze only pins the version of direct, not transitive dependencies. Meaning you don't get deterministic builds.
  - CrLf 6y ago
    What do you mean? If you run pip freeze inside a venv it will print every user-installed module and their dependencies. When installing them in a new venv as a requirements file, you end up with the exact same set of modules.
    - gjvc 6y ago
      Agree. Works fine for me exactly as you describe. I don't understand why people furiously want more than this for the common case. (Choose python version, create .venv/ directory, install packages, use installed .venv/ and do work.)
      - lazypenguin 6y ago
        What happens when you use a different version of the same library that a dependency uses?
        6y ago

gjvc 6y ago

aka UNIX maxim of tools which do one (and only one) thing, and do it well