4 ms·

First of all some years ago pip wasn't really solving dependencies. Then even with the new dependency resolving improvements it's way worse than poetry's in my

First of all some years ago pip wasn't really solving dependencies. Then even with the new dependency resolving improvements it's way worse than poetry's in my experience. Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file. And finally the user interface is much more modern. `poetry shell` is way simpler than `source venv/bin/activate`. You don't even need to run `poetry shell` because there's `poetry run`. And to install a new project is just `poetry install` instead of `python3 -m venv venv; source venv/bin/activate; pip install -r requirements.txt`. Not to mention that it's way easier for me to understand where the depenency conflicts from console output compared to even the newest pip versions.

pen2l 4y ago
Are you guys actually using either poetry or venv on production? Separately, I'm curious to hear examples of where dependency-clutter actually caused a problem.
- plonk 4y ago
  Venv, yes. My Dockerfile's build stage creates a venv and installs all our dependencies (some of which need compilers because they are source wheels that need to link with C++ libraries). Then I copy the venv over to the release stage and install only the release dependencies. Maybe that could work with a `pip --user` install, but at least the venv guarantees that all Python packages and data will be in one self-contained folder.
- butisaidsudo 4y ago
  We use poetry just for development, but run Docker containers in prod. When the image gets built we just create a requirements.txt (poetry export --format requirements.txt --output requirements.txt), copy that into the image, and pip install. Because this is built using the poetry lock file, it'll always be exactly the same unless we specifically update something with poetry. I used to work at a place that was just using requirements.txt files that only included our direct dependencies. There was a project that needed updating after not being touched for a couple of years. The requirements.txt didn't change, but when we built the project again, some of the transitive dependencies used a newer version, and a bug was introduced from one of those updates. A bunch of time was wasted tracking down the issue, pinning the old version of the transitive dependency, and figuring out the damage caused by the bug. As a result, the requirements.txt was changed to also include transitive dependencies. We had vulnerability scanning on our code, and it found a severe issue with one of the transitive dependencies, but there wasn't a version of that library with the issue fixed yet. Time was spent looking into this to see how we could be impacted. As it turns out, it was a transitive dependency for a library that we no longer used and removed from the project months ago. When you create your requirements.txt by running pip freeze > requirements.txt, you don't have an easy way of knowing which library requires which transitive dependency. There's ways you can fix this using multiple requirements.txt files, but at that point it's a lot easier to use poetry, especially if you want to keep your development dependencies separate.
camgunz 4y ago
> And finally the user interface is much more modern. I hear you on the pinning, but "UX" concerns like this drive me batty; I just use shell functions for this: venv() { . ${VIRTUALENV_FOLDER}/${1}/bin/activate } gitsu() { branch=$(git status | head -n 1 | cut -d ' ' -f 3) git push --set-upstream origin ${branch} } git_add_conflicts() { git add $(git status | grep 'both modified:' | cut -d ':' -f2) } It's so much easier to add a little function in your shell than to write a new tool.
- solarkraft 4y ago
  But I would prefer something that just works without me having to do that. Chances are it'll also save me work in areas I'm not aware of yet.
  - camgunz 4y ago
    Oh me too, I love when a tool solves problems I didn't even know I had, until I have them, I look for a solution, and poof Vim (or whatever) has already solved it in a coherent way. If every tool were like that that would be one kind of world, but we live in one that's more interesting. There's some Mark Twain quite like "the world owes you nothing; it was here first", or maybe the Buddha saying "it's harder to soften the earth than it is to wear sandals". I like to think that sandal-making is a core engineering skill, whether you're a carpenter or a computer engineer.
ptx 4y ago
> Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file. This has now been solved [1] in pip, so no third-party tool is needed. Put your pinned dependencies in constraints.txt and your top-level dependencies plus the line "-c constraints.txt" in requirements.txt. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
- partido3619463 4y ago
  Constraints are not the same thing. Poetry locks the entire dependency closure. You can install with “no-deps” option, but then you have to specify specific versions of every dependency in requirements.txt
  - ptx 4y ago
    What do you mean? If you do "pip freeze >constraints.txt", that locks the versions of all installed packages, no matter where they came from. As an example, let's create a venv and install some older versions of Django and its dependencies (current versions are 0.4.2, 3.5.2 and 4.0.6) $ python3 -m venv env1 $ ./env1/bin/pip install sqlparse==0.4.0 asgiref==3.5.0 django=4.0.0 ...and also Flask just to complicate the constraints file for the example: $ ./env1/bin/pip install flask Lock all dependency versions in constraints.txt: $ ./env1/bin/pip freeze >constraints.txt Create a requirements file that specifies just "django" and references the constraints file: $ echo '-c constraints.txt' >requirements.txt $ echo 'django' >>requirements.txt $ cat requirements.txt -c constraints.txt django $ cat constraints.txt asgiref==3.5.0 click==8.1.3 Django==4.0 Flask==2.1.3 itsdangerous==2.1.2 Jinja2==3.1.2 MarkupSafe==2.1.1 sqlparse==0.4.0 Werkzeug==2.1.2 Now we can create another venv with the exact same versions of Django and all its dependencies (but not Flask or its dependencies) using just pip and the requirements file: $ python3 -m venv env2 $ ./env2/bin/pip install -r requirements.txt $ ./env2/bin/pip freeze asgiref==3.5.0 Django==4.0 sqlparse==0.4.0

pen2l 4y ago

Are you guys actually using either poetry or venv on production? Separately, I'm curious to hear examples of where dependency-clutter actually caused a problem.

plonk 4y ago
Venv, yes. My Dockerfile's build stage creates a venv and installs all our dependencies (some of which need compilers because they are source wheels that need to link with C++ libraries). Then I copy the venv over to the release stage and install only the release dependencies. Maybe that could work with a `pip --user` install, but at least the venv guarantees that all Python packages and data will be in one self-contained folder.
butisaidsudo 4y ago
We use poetry just for development, but run Docker containers in prod. When the image gets built we just create a requirements.txt (poetry export --format requirements.txt --output requirements.txt), copy that into the image, and pip install. Because this is built using the poetry lock file, it'll always be exactly the same unless we specifically update something with poetry. I used to work at a place that was just using requirements.txt files that only included our direct dependencies. There was a project that needed updating after not being touched for a couple of years. The requirements.txt didn't change, but when we built the project again, some of the transitive dependencies used a newer version, and a bug was introduced from one of those updates. A bunch of time was wasted tracking down the issue, pinning the old version of the transitive dependency, and figuring out the damage caused by the bug. As a result, the requirements.txt was changed to also include transitive dependencies. We had vulnerability scanning on our code, and it found a severe issue with one of the transitive dependencies, but there wasn't a version of that library with the issue fixed yet. Time was spent looking into this to see how we could be impacted. As it turns out, it was a transitive dependency for a library that we no longer used and removed from the project months ago. When you create your requirements.txt by running pip freeze > requirements.txt, you don't have an easy way of knowing which library requires which transitive dependency. There's ways you can fix this using multiple requirements.txt files, but at that point it's a lot easier to use poetry, especially if you want to keep your development dependencies separate.

camgunz 4y ago

> And finally the user interface is much more modern. I hear you on the pinning, but "UX" concerns like this drive me batty; I just use shell functions for this: venv() { . ${VIRTUALENV_FOLDER}/${1}/bin/activate } gitsu() { branch=$(git status | head -n 1 | cut -d ' ' -f 3) git push --set-upstream origin ${branch} } git_add_conflicts() { git add $(git status | grep 'both modified:' | cut -d ':' -f2) } It's so much easier to add a little function in your shell than to write a new tool.

solarkraft 4y ago
But I would prefer something that just works without me having to do that. Chances are it'll also save me work in areas I'm not aware of yet.
- camgunz 4y ago
  Oh me too, I love when a tool solves problems I didn't even know I had, until I have them, I look for a solution, and poof Vim (or whatever) has already solved it in a coherent way. If every tool were like that that would be one kind of world, but we live in one that's more interesting. There's some Mark Twain quite like "the world owes you nothing; it was here first", or maybe the Buddha saying "it's harder to soften the earth than it is to wear sandals". I like to think that sandal-making is a core engineering skill, whether you're a carpenter or a computer engineer.

ptx 4y ago

> Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file. This has now been solved [1] in pip, so no third-party tool is needed. Put your pinned dependencies in constraints.txt and your top-level dependencies plus the line "-c constraints.txt" in requirements.txt. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files

partido3619463 4y ago
Constraints are not the same thing. Poetry locks the entire dependency closure. You can install with “no-deps” option, but then you have to specify specific versions of every dependency in requirements.txt
- ptx 4y ago
  What do you mean? If you do "pip freeze >constraints.txt", that locks the versions of all installed packages, no matter where they came from. As an example, let's create a venv and install some older versions of Django and its dependencies (current versions are 0.4.2, 3.5.2 and 4.0.6) $ python3 -m venv env1 $ ./env1/bin/pip install sqlparse==0.4.0 asgiref==3.5.0 django=4.0.0 ...and also Flask just to complicate the constraints file for the example: $ ./env1/bin/pip install flask Lock all dependency versions in constraints.txt: $ ./env1/bin/pip freeze >constraints.txt Create a requirements file that specifies just "django" and references the constraints file: $ echo '-c constraints.txt' >requirements.txt $ echo 'django' >>requirements.txt $ cat requirements.txt -c constraints.txt django $ cat constraints.txt asgiref==3.5.0 click==8.1.3 Django==4.0 Flask==2.1.3 itsdangerous==2.1.2 Jinja2==3.1.2 MarkupSafe==2.1.1 sqlparse==0.4.0 Werkzeug==2.1.2 Now we can create another venv with the exact same versions of Django and all its dependencies (but not Flask or its dependencies) using just pip and the requirements file: $ python3 -m venv env2 $ ./env2/bin/pip install -r requirements.txt $ ./env2/bin/pip freeze asgiref==3.5.0 Django==4.0 sqlparse==0.4.0