18 ms·
Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards co
by sjducb 2y ago
Two problems intersect:
- You can’t have two versions of the same package in the namespace at the same time.
- The Python ecosystem is very bad at backwards compatibility
This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above.
There is no good solution to the above problem.
This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained.
The bad solutions are:
1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them.
2) Rewrite your code to only use one library
3) Update one of the libraries
4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library.
Most of the tooling follows approach 1 or 4
- fragmede 2y agoDisk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/ https://github.com/fragmede/python-wool/
- sjducb 2y agoYou’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions. Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
- fragmede 2y agoYeah that sucks. something like: for venv in $(find ~/projects/ -type d -name 'venv'); do ( source ${venv}/bin/activate pip install --upgrade pip pip install --upgrade package_with_cve deactivate ) & done should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.
- fragmede 2y agoI forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.
- tmnvix 2y agoAfter many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory. pyenv for installing and managing python versions. direnv for managing environments and environment variables (a highly underrated package imo). With those two installed I just include a .envrc file in every project. It looks like this: layout python ~/.pyenv/versions/3.11.0/bin/python3 export VARIABLE1=variable1 export VARIABLE2=variable2 etc...
- fragmede 2y agoOh nice. Stick PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages" or whatever in .envrc and Bob's your uncle.