5 ms·

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a depende

by slt2021 2y ago

I see a lot of package managers I never head of, am a happy venv & pip user. One of the key faults of pip is what happens when you decide to remove a dependency. Removing a dependency does not actually remove the sub-dependencies that were brought in by the original dependency, leaving a lot of potential cruft. This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need. This is similar approach to “zero-based budgeting”. It forces you to carefully pick your dependencies and think about what you carry. I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade If you dont do that, you can quickly go down the javascript’s path of bloated node_modules. Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? Even venv is not really required if you dockerize your python apps, which you will have to do anyways at deploy time

BerislavLopac 2y ago
> Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? It's very simple: 1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.
krapht 2y ago
Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems. Conda solves this pain point by being a general purpose package manager. Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.
sealedservant 2y ago
Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.
- doctorpangloss 2y ago
  > There's no concept of caching venv packages elsewhere A venv's pip shares a systemwide cache for downloads.
  - sealedservant 2y ago
    Ah, that's news to me! Updated previous comment for accuracy. I mostly use pipx these days for stuff I need on the PATH.
- slt2021 2y ago
  Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments Never really had an issue with python apps not being portable. Building regular python app? Just use official python docker image. Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt
  - OutOfHere 2y ago
    I dockerize, but using devcontainer.json, and this also plays well with VScode.
- Scaevolus 2y ago
  Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.
  - hadlock 2y ago
    Time to containerize your pytorch app
- rangerelf 2y ago
imglorp 2y ago
> Just nuke and recreate venv from scratch using only what you need. That's great until you have dynamic dependencies which means you can run apparently happy yet fail later on an import. This means you have to take special care generating your dep list.
- slt2021 2y ago
  Thats what pytest is for, it is important to have tests that cover your most important code paths so these these dynamic imports are ran. Due to dynamic natire of language, coding in python without strong test culture is masochism and full of footguns anyways
lexicality 2y ago
I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no dependencies 2. Destroy your virtual environment and reinstall it every day for the next 2 years and fix anything that breaks 3. Freeze your entire dependency tree where it stands and occasionally do large breaking change updates All 3 options suck but #3 is the least worst
- turndown 2y ago
  Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this
  - lexicality 2y ago
    It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees: > I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade
  - kelseyfrog 2y ago
    3 is exactly what you're supposed to do. A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies. The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.
mschuster91 2y ago
> This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need. There was (or still is, I wouldn't be surprised!) a time when the AWS CLI and the AWS Elastic Beanstalk CLI had incompatible versions of some dependency and couldn't be installed in the same venv. Not fun.
abdullahkhalids 2y ago
I use the same workflow, and works for me very well. It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.
- slt2021 2y ago
  I actually would love something like `pip generate-reqs` where pip goes through every .py file in my app and looks at whats being imported and writes these libraries to requirements.txt, together with all transitive dependencies from the currently installed `pip list`
- kiney 2y ago
  pip freeze Is what you're looking for
  - abdullahkhalids 2y ago
    pip freeze lists all possible installed packages with exact version. I want only the manually installed packages in exactly the way I installed them. So pip install numpy>=3.2 should put numpy>=3.2 in requirements.txt. and pip install numpy should put numpy in requirements.txt.
LarsDu88 2y ago
I go over the downsides in the article, but there is nothing fundamentally wrong with using pip and venv. It's just that if you've ever worked in other programming ecosystems, it would be immediately apparent that things could be significantly simpler and more reproducible. Why does python need virtual environments? Why isn't it that you simply are in the correct environment when you're in your project folder? Why are there upwards of a dozen different config files for a project rather than just one standardized one? Why do you need to nuke your whole environment and recreate it just to remove one package? These things are silly. They get in the way of reproducibility. They are security liabilities. There is a better way
- eesmith 2y ago
  > Why isn't it that you simply are in the correct environment when you're in your project folder? I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version for one of my third-party optional dependencies, plus a 3.8 version since I still support that version in my core system. In that way I can quickly test code against the typical deployment cases before committing. The project has about 500K lines of C code, mostly in one Python/C extension which takes over a minute to fully rebuild, but most changes are to Python code or other bits of Python/C or Cython code which are faster to rebuild, so I do editable builds for most of my work for the quick edit cycle. I don't know how to nominate one of these environments as "correct". (The full test suite against the various permutations of Python versions and 10 or so optional components is a mess. I've given up on testing all combinations as it wasn't proving worthwhile.)
  - LarsDu88 2y ago
    Sounds like you may want to check out Hatch
    - eesmith 2y ago
      How would that help? At the end I'm still using setuptools to build the extension, right? So I'm stuck with the same core issues, but with a different front end? To add to the complexity, I distribute my software as source, and I am wary of telling my customers to install another package in order to run the commands to install my package.
jamesboehmer 2y ago
pip is so easy, but unfortunately i've found that if you add package signatures to requirements.txt, pip chokes on it when installing it later. And subdependencies aren't always named perfectly, e.g. they might specify ~=1.4, and a subdependency that what was once 1.4.0 is now 1.4.27, and incompatible or compromised. conda is so heavyweight installing whole pre-approved builds. and the command line options I find extremely frustrating. I need supply chain security and perfectly reproducible builds, so poetry was the only real option.
- slt2021 2y ago
  wouldn't `pip freeze > requirements.txt` solve your problem? It will list everything currently installed, including transitive dependencies at currently installed and working versions
darepublic 2y ago
I also use venv and pip and I'm happy with the solution. I don't get the knock on JavaScript node modules though. Why wouldn't you want to list out your dependencies and their versions? It can be greatly helpful and I don't see how it contributes to bloat
eternityforest 2y ago
Pulling the latest version without freezing is how you Break Stuff. Every python developer seems to be addicted to breaking changes. Nuking and recreating, freezing, etc are all easy minor tasks but nonetheless add a few lines each of code. Auditing for security issues would probably need some other package or other. And with virtualenv, you can have an activated environment. It's stateful and your scripts have to then take into account state. Poetry run doesn't have that issue. That's enough minor annoyances to make me not want to ever use anything but poetry plus the audit plugin and freeze-wheel if needed.
alexey-salmin 2y ago
> I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade This gets you into situations when you need to release an urgent one-line hotfix into production but your docker builds are broken because some version changed. Stable reproducible build are an absolute must if you ask me.

BerislavLopac 2y ago

> Can people explain why venv&pip is a bad solution that doesnt work for them they have to resort to other package managers? It's very simple: 1. As you said yourself, you don't need venv at all. 2. The format used by `pip -r` (colloquially known as "requirements file") is not a defined standard. 3. pip is not a package manager, but a package version resolver and installer.

krapht 2y ago

Pip is mediocre for vendoring non-Python or binary dependencies. Wheels are OK, but sometimes there are still problems. Conda solves this pain point by being a general purpose package manager. Poetry is for the more reproducible builds people. It's a little better, but it's not great enough to get people to switch.

sealedservant 2y ago

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.

doctorpangloss 2y ago
> There's no concept of caching venv packages elsewhere A venv's pip shares a systemwide cache for downloads.
- sealedservant 2y ago
  Ah, that's news to me! Updated previous comment for accuracy. I mostly use pipx these days for stuff I need on the PATH.
slt2021 2y ago
Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments Never really had an issue with python apps not being portable. Building regular python app? Just use official python docker image. Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt
- OutOfHere 2y ago
  I dockerize, but using devcontainer.json, and this also plays well with VScode.
Scaevolus 2y ago
Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.
- hadlock 2y ago
  Time to containerize your pytorch app
rangerelf 2y ago

imglorp 2y ago

> Just nuke and recreate venv from scratch using only what you need. That's great until you have dynamic dependencies which means you can run apparently happy yet fail later on an import. This means you have to take special care generating your dep list.

slt2021 2y ago
Thats what pytest is for, it is important to have tests that cover your most important code paths so these these dynamic imports are ran. Due to dynamic natire of language, coding in python without strong test culture is masochism and full of footguns anyways

lexicality 2y ago

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no dependencies 2. Destroy your virtual environment and reinstall it every day for the next 2 years and fix anything that breaks 3. Freeze your entire dependency tree where it stands and occasionally do large breaking change updates All 3 options suck but #3 is the least worst

turndown 2y ago
Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this
- lexicality 2y ago
  It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees: > I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade
- kelseyfrog 2y ago
  3 is exactly what you're supposed to do. A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies. The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.

mschuster91 2y ago

> This is not really an issue if your virtual environments are disposable. Just nuke and recreate venv from scratch using only what you need. There was (or still is, I wouldn't be surprised!) a time when the AWS CLI and the AWS Elastic Beanstalk CLI had incompatible versions of some dependency and couldn't be installed in the same venv. Not fun.

abdullahkhalids 2y ago

I use the same workflow, and works for me very well. It would be cool, though, to have a wrapper around pip that adds any packages I install to the requirements file. That way I don't have to do it manually.

slt2021 2y ago
I actually would love something like `pip generate-reqs` where pip goes through every .py file in my app and looks at whats being imported and writes these libraries to requirements.txt, together with all transitive dependencies from the currently installed `pip list`
kiney 2y ago
pip freeze Is what you're looking for
- abdullahkhalids 2y ago
  pip freeze lists all possible installed packages with exact version. I want only the manually installed packages in exactly the way I installed them. So pip install numpy>=3.2 should put numpy>=3.2 in requirements.txt. and pip install numpy should put numpy in requirements.txt.

LarsDu88 2y ago

I go over the downsides in the article, but there is nothing fundamentally wrong with using pip and venv. It's just that if you've ever worked in other programming ecosystems, it would be immediately apparent that things could be significantly simpler and more reproducible. Why does python need virtual environments? Why isn't it that you simply are in the correct environment when you're in your project folder? Why are there upwards of a dozen different config files for a project rather than just one standardized one? Why do you need to nuke your whole environment and recreate it just to remove one package? These things are silly. They get in the way of reproducibility. They are security liabilities. There is a better way

eesmith 2y ago
> Why isn't it that you simply are in the correct environment when you're in your project folder? I typically have several environments for my main project. Mostly I use a Python 3.12 version with the full set of optional third-party packages installed, but I also have a "clean" version with none of them installed so I can test error handling, as well as a Python 3.10 version as that is the oldest supported version for one of my third-party optional dependencies, plus a 3.8 version since I still support that version in my core system. In that way I can quickly test code against the typical deployment cases before committing. The project has about 500K lines of C code, mostly in one Python/C extension which takes over a minute to fully rebuild, but most changes are to Python code or other bits of Python/C or Cython code which are faster to rebuild, so I do editable builds for most of my work for the quick edit cycle. I don't know how to nominate one of these environments as "correct". (The full test suite against the various permutations of Python versions and 10 or so optional components is a mess. I've given up on testing all combinations as it wasn't proving worthwhile.)
- LarsDu88 2y ago
  Sounds like you may want to check out Hatch
  - eesmith 2y ago
    How would that help? At the end I'm still using setuptools to build the extension, right? So I'm stuck with the same core issues, but with a different front end? To add to the complexity, I distribute my software as source, and I am wary of telling my customers to install another package in order to run the commands to install my package.

jamesboehmer 2y ago

pip is so easy, but unfortunately i've found that if you add package signatures to requirements.txt, pip chokes on it when installing it later. And subdependencies aren't always named perfectly, e.g. they might specify ~=1.4, and a subdependency that what was once 1.4.0 is now 1.4.27, and incompatible or compromised. conda is so heavyweight installing whole pre-approved builds. and the command line options I find extremely frustrating. I need supply chain security and perfectly reproducible builds, so poetry was the only real option.

slt2021 2y ago
wouldn't `pip freeze > requirements.txt` solve your problem? It will list everything currently installed, including transitive dependencies at currently installed and working versions

darepublic 2y ago

I also use venv and pip and I'm happy with the solution. I don't get the knock on JavaScript node modules though. Why wouldn't you want to list out your dependencies and their versions? It can be greatly helpful and I don't see how it contributes to bloat

eternityforest 2y ago

Pulling the latest version without freezing is how you Break Stuff. Every python developer seems to be addicted to breaking changes. Nuking and recreating, freezing, etc are all easy minor tasks but nonetheless add a few lines each of code. Auditing for security issues would probably need some other package or other. And with virtualenv, you can have an activated environment. It's stateful and your scripts have to then take into account state. Poetry run doesn't have that issue. That's enough minor annoyances to make me not want to ever use anything but poetry plus the audit plugin and freeze-wheel if needed.

alexey-salmin 2y ago

> I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade This gets you into situations when you need to release an urgent one-line hotfix into production but your docker builds are broken because some version changed. Stable reproducible build are an absolute must if you ask me.