5 ms·

Do people doing ML/DS not use conda anymore?

buildbot 2y ago
A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389 https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison. I'm an Applied Scientist vs. ML Engineer, if that matters.
- astromaniak 2y ago
  It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler... Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.
  - buildbot 2y ago
    I do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.
copperroof 2y ago
If they are they should stop. It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community. Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.
- buildbot 2y ago
  Fully agree with this. Understand the basic tools that currently exist and you'll be fine. Conda constantly fucks things up in weird hard to debug ways...
blt 2y ago
I can't point at a single reason, but I got sick of it. The interminable solves were awful. Mamba made it better, but can still be slow. Plenty of more esoteric packages are on PyPI but not Conda. (Yes, you can install pip packages in a conda env file.) Many packages have a default version and a conda-forge version; it's not always clear which you should use. In Github CI, it takes extra time to install. Upon installation it (by default) wants to mess around with your .bashrc and start every shell in a "base" environment. It's operated by another company instead of the Python Software Foundation. idk, none of these are deal-breakers, but I switched to venv and have not considered going back.

A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389 https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison. I'm an Applied Scientist vs. ML Engineer, if that matters.

astromaniak 2y ago
It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler... Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.
- buildbot 2y ago
  I do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.

copperroof 2y ago

If they are they should stop. It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community. Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.

buildbot 2y ago
Fully agree with this. Understand the basic tools that currently exist and you'll be fine. Conda constantly fucks things up in weird hard to debug ways...

blt 2y ago

I can't point at a single reason, but I got sick of it. The interminable solves were awful. Mamba made it better, but can still be slow. Plenty of more esoteric packages are on PyPI but not Conda. (Yes, you can install pip packages in a conda env file.) Many packages have a default version and a conda-forge version; it's not always clear which you should use. In Github CI, it takes extra time to install. Upon installation it (by default) wants to mess around with your .bashrc and start every shell in a "base" environment. It's operated by another company instead of the Python Software Foundation. idk, none of these are deal-breakers, but I switched to venv and have not considered going back.