9 ms·

I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are thes

I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but I've never come across it as a python dev, and the moment I try to user/understand these tools I feel like they're against the "keep it simple, stupid" vibe that python gives me. There's auto-venv for the truly lazy (like me) but other than that I don't rely on any of these Env/requirement wrappers or tools in any of my projects (virtualenv-wrapper for work but that was in their setup guide). Is it a legacy thing?

throwaway234232 4y ago
You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.
- traverseda 4y ago
  I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly. If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program. Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.
  - throwaway234232 4y ago
    I agree, but they also include hashes.
  - mirashii 4y ago
    Semver promises are insufficient in a world where supply chain attacks are increasingly common. Pulling untested and invalidated code in at every project build is how that transitive dependency on a package that was taken over for a small window wrecks your development team. You should never be pulling in new code by surprise, it should always be something I’m aware of and signing up for. The Rust ecosystem is good evidence that locking doesn’t kill semver. Semver is still widely used and has all of its meaning.
  - remram 4y ago
    > You shouldn't need to use lock files if you're using semver properly. You mean "if all your dependencies are using semver properly". Yes, you're right, and you've found the problem.
  - nickjj 4y ago
    > You shouldn't need to use lock files if you're using semver properly. I think this issue is unrelated to semver. I've had so many issues over the years with Python packages, even very polished libraries like Flask and Celery. For example, you could install something like Flask 2.0 but end up with a major difference in versions of its sub-dependencies depending on when you installed it. That's because Flask has its own dependencies defined like this: "Werkzeug >= 2.2.0a1", "Jinja2 >= 3.0", "itsdangerous >= 2.0", "click >= 8.0", The above means installing Flask 2.0 could one day in the future install Jinja 4 or Click 10 unless you lock your entire dependency tree. I've also had all sorts of things break because Flask installed Jinja 3.1 which wasn't a problem 6 months ago when Jinja 3.0 was the latest release. I've also had cases where installing a specific version of Celery in the past worked but failed in the future because it didn't lock one of its sub-dependencies down well enough (vine) which caused a breaking change. This stuff happens all the time and it's a nuisance. I never experienced issues like this with Ruby and other languages that have the idea of a lock file built into their package manager. IMO it's a desperately needed feature that should be built into pip.
  - camgunz 4y ago
    I've experienced multiple minor and patch (according to semver) updates that broke APIs and behavior, and I'd guess most devs have as well. I think semver makes sense to humans. I can derive a lot of meaning from it when I see n.n.n. But when it comes to the software supply chain, it's just too rickety. Frankly, when you lose customers after a new deploy broke one of your dependencies, "but the dependency author didn't respect semver" isn't an excuse. I say this as someone who strongly pushed `dependency>=1.3.2,<1.4` until that happened to me. My argument was "security updates", and now I just don't care. The software supply chain is too chaotic, and you have to be defensive against it.
- hibbelig 4y ago
  The difference between virtual environments in Python and node_modules in JavaScript is minor I would say. It’s the same concept. I believe composer has something similar. It’s a directory where packages go and they are not part of the system wide installation. I agree that lock files are useful and it’s a pity that pip does not offer them.
plonk 4y ago
I just use venv but don’t fix dependencies. We often develop new packages and fixing every dependency’s version sounds like unnecessary effort. With sensibly written setup.cfg files, we just “pip install” all our packages with no issues. Pip’s dependency resolver has come a long way since 2019.
jks 4y ago
The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.
- julienpalard 4y ago
  Upgrading dependencies is also a way for me to resolve issues, so I don't fight it.
- bvrmn 4y ago
  pip freeze > stable-reqs.txt ??
  - throwaway234232 4y ago
    https://github.com/pypa/pip/issues/4732 https://github.com/pypa/pip/issues/4732
    - camgunz 4y ago
      This thread is just about pip freeze not including hashes, and while it's true that would provide a lot of reassurance, you can still pin transitive version dependencies w/o them.
  - potatochup 4y ago
    This kinda works, but it isn't a cross-platform solution.
- traverseda 4y ago
  That's a feature, you should be using the latest minor/patch release when developing. If you really need to lock a dependency to a specific version it is possible, but you shouldn't be doing it. If a dependency is constantly breaking on minor/patch releases you shouldn't be using it. I think it's a lot of javascript developers who end up having this problem, and all I can say is that the python package ecosystem is not the same. Please don't lock your dependencies to specific minor/patch versions, and please don't use so many dependencies that it becomes tedious to deal with changes. Especially when you're writing a library. If you're locking to anything other than a major release or a minimum minor/patch revision when producing a library things have gone very wrong.
DanielVZ 4y ago
First of all some years ago pip wasn't really solving dependencies. Then even with the new dependency resolving improvements it's way worse than poetry's in my experience. Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file. And finally the user interface is much more modern. `poetry shell` is way simpler than `source venv/bin/activate`. You don't even need to run `poetry shell` because there's `poetry run`. And to install a new project is just `poetry install` instead of `python3 -m venv venv; source venv/bin/activate; pip install -r requirements.txt`. Not to mention that it's way easier for me to understand where the depenency conflicts from console output compared to even the newest pip versions.
- pen2l 4y ago
  Are you guys actually using either poetry or venv on production? Separately, I'm curious to hear examples of where dependency-clutter actually caused a problem.
  - plonk 4y ago
    Venv, yes. My Dockerfile's build stage creates a venv and installs all our dependencies (some of which need compilers because they are source wheels that need to link with C++ libraries). Then I copy the venv over to the release stage and install only the release dependencies. Maybe that could work with a `pip --user` install, but at least the venv guarantees that all Python packages and data will be in one self-contained folder.
  - butisaidsudo 4y ago
    We use poetry just for development, but run Docker containers in prod. When the image gets built we just create a requirements.txt (poetry export --format requirements.txt --output requirements.txt), copy that into the image, and pip install. Because this is built using the poetry lock file, it'll always be exactly the same unless we specifically update something with poetry. I used to work at a place that was just using requirements.txt files that only included our direct dependencies. There was a project that needed updating after not being touched for a couple of years. The requirements.txt didn't change, but when we built the project again, some of the transitive dependencies used a newer version, and a bug was introduced from one of those updates. A bunch of time was wasted tracking down the issue, pinning the old version of the transitive dependency, and figuring out the damage caused by the bug. As a result, the requirements.txt was changed to also include transitive dependencies. We had vulnerability scanning on our code, and it found a severe issue with one of the transitive dependencies, but there wasn't a version of that library with the issue fixed yet. Time was spent looking into this to see how we could be impacted. As it turns out, it was a transitive dependency for a library that we no longer used and removed from the project months ago. When you create your requirements.txt by running pip freeze > requirements.txt, you don't have an easy way of knowing which library requires which transitive dependency. There's ways you can fix this using multiple requirements.txt files, but at that point it's a lot easier to use poetry, especially if you want to keep your development dependencies separate.
raverbashing 4y ago
I think the same Some tools, like poetry do add value to your development process (namely, better management of dependencies) pipenv really does sound like something for people that are too lazy to activate their envs before running stuff
deleted 4y ago
[deleted]
Macha 4y ago
What do you list in your requirements.txt? If it's just the top level dependencies, you could have versions change of your transitive dependencies which can matter and break things. If it's all dependencies, you lose the context of what you need because you use it and what you need just because it's a dependency of a dependency. Some projects used to have two files for this, effectively managing a lockfile manually with pip freeze, but then it's nice to have a wrapper around this pattern and that's where the first gen like pip-tools came from, and then stuff like poetry/pipenv is aiming to streamline that (and avoid manual use of companions like pyenv for specific Python versons) even more.
smetj 4y ago
Totally agree on this.

throwaway234232 4y ago

You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.

traverseda 4y ago
I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly. If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program. Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.
- throwaway234232 4y ago
  I agree, but they also include hashes.
- mirashii 4y ago
  Semver promises are insufficient in a world where supply chain attacks are increasingly common. Pulling untested and invalidated code in at every project build is how that transitive dependency on a package that was taken over for a small window wrecks your development team. You should never be pulling in new code by surprise, it should always be something I’m aware of and signing up for. The Rust ecosystem is good evidence that locking doesn’t kill semver. Semver is still widely used and has all of its meaning.
- remram 4y ago
  > You shouldn't need to use lock files if you're using semver properly. You mean "if all your dependencies are using semver properly". Yes, you're right, and you've found the problem.
- nickjj 4y ago
  > You shouldn't need to use lock files if you're using semver properly. I think this issue is unrelated to semver. I've had so many issues over the years with Python packages, even very polished libraries like Flask and Celery. For example, you could install something like Flask 2.0 but end up with a major difference in versions of its sub-dependencies depending on when you installed it. That's because Flask has its own dependencies defined like this: "Werkzeug >= 2.2.0a1", "Jinja2 >= 3.0", "itsdangerous >= 2.0", "click >= 8.0", The above means installing Flask 2.0 could one day in the future install Jinja 4 or Click 10 unless you lock your entire dependency tree. I've also had all sorts of things break because Flask installed Jinja 3.1 which wasn't a problem 6 months ago when Jinja 3.0 was the latest release. I've also had cases where installing a specific version of Celery in the past worked but failed in the future because it didn't lock one of its sub-dependencies down well enough (vine) which caused a breaking change. This stuff happens all the time and it's a nuisance. I never experienced issues like this with Ruby and other languages that have the idea of a lock file built into their package manager. IMO it's a desperately needed feature that should be built into pip.
- camgunz 4y ago
  I've experienced multiple minor and patch (according to semver) updates that broke APIs and behavior, and I'd guess most devs have as well. I think semver makes sense to humans. I can derive a lot of meaning from it when I see n.n.n. But when it comes to the software supply chain, it's just too rickety. Frankly, when you lose customers after a new deploy broke one of your dependencies, "but the dependency author didn't respect semver" isn't an excuse. I say this as someone who strongly pushed `dependency>=1.3.2,<1.4` until that happened to me. My argument was "security updates", and now I just don't care. The software supply chain is too chaotic, and you have to be defensive against it.
hibbelig 4y ago
The difference between virtual environments in Python and node_modules in JavaScript is minor I would say. It’s the same concept. I believe composer has something similar. It’s a directory where packages go and they are not part of the system wide installation. I agree that lock files are useful and it’s a pity that pip does not offer them.

plonk 4y ago

I just use venv but don’t fix dependencies. We often develop new packages and fixing every dependency’s version sounds like unnecessary effort. With sensibly written setup.cfg files, we just “pip install” all our packages with no issues. Pip’s dependency resolver has come a long way since 2019.

jks 4y ago

The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.

julienpalard 4y ago
Upgrading dependencies is also a way for me to resolve issues, so I don't fight it.
bvrmn 4y ago
pip freeze > stable-reqs.txt ??
- throwaway234232 4y ago
  https://github.com/pypa/pip/issues/4732 https://github.com/pypa/pip/issues/4732
  - camgunz 4y ago
    This thread is just about pip freeze not including hashes, and while it's true that would provide a lot of reassurance, you can still pin transitive version dependencies w/o them.
- potatochup 4y ago
  This kinda works, but it isn't a cross-platform solution.
traverseda 4y ago
That's a feature, you should be using the latest minor/patch release when developing. If you really need to lock a dependency to a specific version it is possible, but you shouldn't be doing it. If a dependency is constantly breaking on minor/patch releases you shouldn't be using it. I think it's a lot of javascript developers who end up having this problem, and all I can say is that the python package ecosystem is not the same. Please don't lock your dependencies to specific minor/patch versions, and please don't use so many dependencies that it becomes tedious to deal with changes. Especially when you're writing a library. If you're locking to anything other than a major release or a minimum minor/patch revision when producing a library things have gone very wrong.

DanielVZ 4y ago

First of all some years ago pip wasn't really solving dependencies. Then even with the new dependency resolving improvements it's way worse than poetry's in my experience. Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file. And finally the user interface is much more modern. `poetry shell` is way simpler than `source venv/bin/activate`. You don't even need to run `poetry shell` because there's `poetry run`. And to install a new project is just `poetry install` instead of `python3 -m venv venv; source venv/bin/activate; pip install -r requirements.txt`. Not to mention that it's way easier for me to understand where the depenency conflicts from console output compared to even the newest pip versions.

pen2l 4y ago
Are you guys actually using either poetry or venv on production? Separately, I'm curious to hear examples of where dependency-clutter actually caused a problem.
- plonk 4y ago
  Venv, yes. My Dockerfile's build stage creates a venv and installs all our dependencies (some of which need compilers because they are source wheels that need to link with C++ libraries). Then I copy the venv over to the release stage and install only the release dependencies. Maybe that could work with a `pip --user` install, but at least the venv guarantees that all Python packages and data will be in one self-contained folder.
- butisaidsudo 4y ago
  We use poetry just for development, but run Docker containers in prod. When the image gets built we just create a requirements.txt (poetry export --format requirements.txt --output requirements.txt), copy that into the image, and pip install. Because this is built using the poetry lock file, it'll always be exactly the same unless we specifically update something with poetry. I used to work at a place that was just using requirements.txt files that only included our direct dependencies. There was a project that needed updating after not being touched for a couple of years. The requirements.txt didn't change, but when we built the project again, some of the transitive dependencies used a newer version, and a bug was introduced from one of those updates. A bunch of time was wasted tracking down the issue, pinning the old version of the transitive dependency, and figuring out the damage caused by the bug. As a result, the requirements.txt was changed to also include transitive dependencies. We had vulnerability scanning on our code, and it found a severe issue with one of the transitive dependencies, but there wasn't a version of that library with the issue fixed yet. Time was spent looking into this to see how we could be impacted. As it turns out, it was a transitive dependency for a library that we no longer used and removed from the project months ago. When you create your requirements.txt by running pip freeze > requirements.txt, you don't have an easy way of knowing which library requires which transitive dependency. There's ways you can fix this using multiple requirements.txt files, but at that point it's a lot easier to use poetry, especially if you want to keep your development dependencies separate.

raverbashing 4y ago

I think the same Some tools, like poetry do add value to your development process (namely, better management of dependencies) pipenv really does sound like something for people that are too lazy to activate their envs before running stuff

deleted 4y ago

[deleted]

Macha 4y ago

What do you list in your requirements.txt? If it's just the top level dependencies, you could have versions change of your transitive dependencies which can matter and break things. If it's all dependencies, you lose the context of what you need because you use it and what you need just because it's a dependency of a dependency. Some projects used to have two files for this, effectively managing a lockfile manually with pip freeze, but then it's nice to have a wrapper around this pattern and that's where the first gen like pip-tools came from, and then stuff like poetry/pipenv is aiming to streamline that (and avoid manual use of companions like pyenv for specific Python versons) even more.

smetj 4y ago

Totally agree on this.