4 ms·

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacemen

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are python engineers just bad, or? (Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)

sjducb 2y ago
Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained. The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library. Most of the tooling follows approach 1 or 4
- fragmede 2y ago
  Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/ https://github.com/fragmede/python-wool/
  - sjducb 2y ago
    You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions. Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
    - fragmede 2y ago
      Yeah that sucks. something like: for venv in $(find ~/projects/ -type d -name 'venv'); do ( source ${venv}/bin/activate pip install --upgrade pip pip install --upgrade package_with_cve deactivate ) & done should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.
      - fragmede 2y ago
        I forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.
  - tmnvix 2y ago
    After many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory. pyenv for installing and managing python versions. direnv for managing environments and environment variables (a highly underrated package imo). With those two installed I just include a .envrc file in every project. It looks like this: layout python ~/.pyenv/versions/3.11.0/bin/python3 export VARIABLE1=variable1 export VARIABLE2=variable2 etc...
    - fragmede 2y ago
      Oh nice. Stick PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages" or whatever in .envrc and Bob's your uncle.
fbdab103 2y ago
I think it is worth separating the Python ML ecosystem from the rest. While traditional Python environment management has many sore points, it is usually not terrible (though there are many gotchas still-to-this-day-problems that should have been corrected long ago). The ML system is a whole another stack of problems. The elephant in the room is Nvidia who is not known for playing well with others. Aside from that, the state of the art in ML is churning rapidly as new improvements are identified.
globular-toast 2y ago
It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.
lloydatkinson 2y ago
Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning. I can only imagine it seemed like an oasis to R which is bottom tier. So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses… things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.
reportgunner 2y ago
Just replace python with mac or apple in your comment and I think you will understand.

sjducb 2y ago

Two problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained. The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library. Most of the tooling follows approach 1 or 4

fragmede 2y ago
Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/ https://github.com/fragmede/python-wool/
- sjducb 2y ago
  You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions. Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
  - fragmede 2y ago
    Yeah that sucks. something like: for venv in $(find ~/projects/ -type d -name 'venv'); do ( source ${venv}/bin/activate pip install --upgrade pip pip install --upgrade package_with_cve deactivate ) & done should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.
    - fragmede 2y ago
      I forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.
- tmnvix 2y ago
  After many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory. pyenv for installing and managing python versions. direnv for managing environments and environment variables (a highly underrated package imo). With those two installed I just include a .envrc file in every project. It looks like this: layout python ~/.pyenv/versions/3.11.0/bin/python3 export VARIABLE1=variable1 export VARIABLE2=variable2 etc...
  - fragmede 2y ago
    Oh nice. Stick PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages" or whatever in .envrc and Bob's your uncle.

fbdab103 2y ago

I think it is worth separating the Python ML ecosystem from the rest. While traditional Python environment management has many sore points, it is usually not terrible (though there are many gotchas still-to-this-day-problems that should have been corrected long ago). The ML system is a whole another stack of problems. The elephant in the room is Nvidia who is not known for playing well with others. Aside from that, the state of the art in ML is churning rapidly as new improvements are identified.

globular-toast 2y ago

It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.

lloydatkinson 2y ago

Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning. I can only imagine it seemed like an oasis to R which is bottom tier. So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses… things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.

reportgunner 2y ago

Just replace python with mac or apple in your comment and I think you will understand.