6 ms·

I'm a bit surprised that conda is not mentioned anywhere (I'm not saying as best practice, but as an alternative or why envs are better)

by kuu 5y ago

karxxm 5y ago
Yup, especially when you are working with os-level dependencies like cuda, conda is the way to go. I think every tensorflow user can approve
- codethief 5y ago
  …unless you're working on a project for arm64-based IoT devices (like Nvidia Jetson), where Conda is not available.
  - hantusk 5y ago
    Does it not work with miniforge/mambaforge? (https://github.com/conda-forge/miniforge https://github.com/conda-forge/miniforge) I use this with my m1 mac, and it works great.
  - wdroz 5y ago
    conda is now available[1] for arm64 (since March[2]). [1] -- https://docs.conda.io/en/latest/miniconda.html#linux-installers https://docs.conda.io/en/latest/miniconda.html#linux-install... [2] -- https://github.com/conda/conda/issues/8297 https://github.com/conda/conda/issues/8297
    - codethief 5y ago
      Awww man. I must have missed that announcement by only a few days.
- dijksterhuis 5y ago
  Quite confused about this comment as many of the latest tensorflow V2 releases aren't reliably uploaded to conda forge. IIRC there's only like 4 or so of the V2 releases uploaded. You can conda install the CUDA dependencies and then install the required Tensorflow version via conda pip. But that's not much different to installing CUDA manually and then installing tf from system pip. It's much faster and easier to pull a tf Docker image as it's their "officially supported" way to get up and running. So... as a tf user and a sysadmin... Nah. No conda for me thanks.
  - karxxm 5y ago
    One project uses tf 2.1, one uses 1.15 and the other 2.4 ... for me it is much more convenient to have 3 envs rather than 3 containers or switching the system cuda as needed... I especially had problems debugging through docker containers back in the days, therefore i never picked it up again.
wdroz 5y ago
Yes, if you are doing anything close to machine learning, conda is the most favored tool.
rwmark 5y ago
conda is developed by heretics and not the ideologically sound country club.
laminatedsmore 5y ago
Conda has been a lifesaver for me in the past, but it got so slow in ~2019 (minutes+ to resolve dependencies) that I've switched back to pip whenever possible. Maybe things have been resolved now though? E.g. https://github.com/conda/conda/issues/8087 https://github.com/conda/conda/issues/8087, https://www.anaconda.com/blog/understanding-and-improving-condas-performance https://www.anaconda.com/blog/understanding-and-improving-co...
- sseagull 5y ago
  I have an environment (somewhat complicated but not that bad really) that occasionally takes more than an hour. Oh and if there is a conflict somewhere, it goes into some conflict detection routine that will take hours and not produce anything useful. I could go on, but I have come to really dislike conda.
- lr1970 5y ago
  > Conda has been a lifesaver for me in the past, but it got so slow in ~2019 This is why mamba [0] was created. It is a C++ reimplementation of conda for much better performance. mamba is a drop-in replacement of conda and can operate on the same anaconda, condaforge (and mambaforge) repositories. [0] https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba
  - sseagull 5y ago
    I do have to try mamba sometime but I feel like there is something more than python slowness going on. I use Gentoo and its package manager is written in python. Even though it is more complex (IMO) it doesn’t have nearly the same slowness when it comes to dependency resolution and conflict detection.
gillesjacobs 5y ago
My only problem with conda is the proliferation of package sources and lack of dev support.
matsemann 5y ago
Conda is one of the most annoying things with Python imo. Or, it's just another symptom of the crazy dependency hell of python, but it's actively making it worse. Now every project is a mashup of conda and pip and probably more. Just right now I'm trying to fire up a new instance on GCP. With a completely clean image, doing a conda install hangs for 30 minutes while it's trying to "solve" something.
u678u 5y ago
conda was necessarily 5 years ago but now pip seems a lot more reliable so I've given up using it.
- zo1 5y ago
  Yeah, now all the "hip" devs are driving things towards "poetry". It's very disillusioning to see how sheer twitter-followings and "popularity" type metrics drive development these days by forcing alternatives to be de-facto neglected. Everyone does what's "hot", so all the tutorials and bug reports and tests and SO questions and new libraries and and and all go towards that framework or language or tool or method. You can't even argue technical merits towards the neglected options because yes the popular tool is better, but only because we have a metric boat load (millions) of man-hours being pumped into making it better instead of all the alternatives. It's like the tech-equivalent of fashion fads in that it's self-reinforcing. Not to take away from some of the actual and technical achievements that some of these things have made, of course.
  - u678u 5y ago
    You should see our k8s migration, its the biggest waste of time for everyone I've ever seen.
  - jonnycomputer 5y ago
    I guess. But frankly, all the solutions were than not that great. So its not surprising people might want to move to something new.
KaiserPro 5y ago
Conda is a ginormous pain in the arse. I supported a research cluster, and the amount of times conda caused an issue was a lot.
pmkiwi 5y ago
conda makes sense in some circumstances but you need to invest a lot of times to undersand this tool (remind me my Maven times sometimes...). With all scientist packages, conda is pretty good but it is sooooo slow, it is just insane!

I'm a bit surprised that conda is not mentioned anywhere (I'm not saying as best practice, but as an alternative or why envs are better)

by kuu 5y ago

I'm a bit surprised that conda is not mentioned anywhere (I'm not saying as best practice, but as an alternative or why envs are better)

karxxm 5y ago

Yup, especially when you are working with os-level dependencies like cuda, conda is the way to go. I think every tensorflow user can approve

codethief 5y ago
…unless you're working on a project for arm64-based IoT devices (like Nvidia Jetson), where Conda is not available.
- hantusk 5y ago
  Does it not work with miniforge/mambaforge? (https://github.com/conda-forge/miniforge https://github.com/conda-forge/miniforge) I use this with my m1 mac, and it works great.
- wdroz 5y ago
  conda is now available[1] for arm64 (since March[2]). [1] -- https://docs.conda.io/en/latest/miniconda.html#linux-installers https://docs.conda.io/en/latest/miniconda.html#linux-install... [2] -- https://github.com/conda/conda/issues/8297 https://github.com/conda/conda/issues/8297
  - codethief 5y ago
    Awww man. I must have missed that announcement by only a few days.
dijksterhuis 5y ago
Quite confused about this comment as many of the latest tensorflow V2 releases aren't reliably uploaded to conda forge. IIRC there's only like 4 or so of the V2 releases uploaded. You can conda install the CUDA dependencies and then install the required Tensorflow version via conda pip. But that's not much different to installing CUDA manually and then installing tf from system pip. It's much faster and easier to pull a tf Docker image as it's their "officially supported" way to get up and running. So... as a tf user and a sysadmin... Nah. No conda for me thanks.
- karxxm 5y ago
  One project uses tf 2.1, one uses 1.15 and the other 2.4 ... for me it is much more convenient to have 3 envs rather than 3 containers or switching the system cuda as needed... I especially had problems debugging through docker containers back in the days, therefore i never picked it up again.

wdroz 5y ago

Yes, if you are doing anything close to machine learning, conda is the most favored tool.

rwmark 5y ago

conda is developed by heretics and not the ideologically sound country club.

laminatedsmore 5y ago

Conda has been a lifesaver for me in the past, but it got so slow in ~2019 (minutes+ to resolve dependencies) that I've switched back to pip whenever possible. Maybe things have been resolved now though? E.g. https://github.com/conda/conda/issues/8087 https://github.com/conda/conda/issues/8087, https://www.anaconda.com/blog/understanding-and-improving-condas-performance https://www.anaconda.com/blog/understanding-and-improving-co...

sseagull 5y ago
I have an environment (somewhat complicated but not that bad really) that occasionally takes more than an hour. Oh and if there is a conflict somewhere, it goes into some conflict detection routine that will take hours and not produce anything useful. I could go on, but I have come to really dislike conda.
lr1970 5y ago
> Conda has been a lifesaver for me in the past, but it got so slow in ~2019 This is why mamba [0] was created. It is a C++ reimplementation of conda for much better performance. mamba is a drop-in replacement of conda and can operate on the same anaconda, condaforge (and mambaforge) repositories. [0] https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba
- sseagull 5y ago
  I do have to try mamba sometime but I feel like there is something more than python slowness going on. I use Gentoo and its package manager is written in python. Even though it is more complex (IMO) it doesn’t have nearly the same slowness when it comes to dependency resolution and conflict detection.

gillesjacobs 5y ago

My only problem with conda is the proliferation of package sources and lack of dev support.

matsemann 5y ago

Conda is one of the most annoying things with Python imo. Or, it's just another symptom of the crazy dependency hell of python, but it's actively making it worse. Now every project is a mashup of conda and pip and probably more. Just right now I'm trying to fire up a new instance on GCP. With a completely clean image, doing a conda install hangs for 30 minutes while it's trying to "solve" something.

u678u 5y ago

conda was necessarily 5 years ago but now pip seems a lot more reliable so I've given up using it.

zo1 5y ago
Yeah, now all the "hip" devs are driving things towards "poetry". It's very disillusioning to see how sheer twitter-followings and "popularity" type metrics drive development these days by forcing alternatives to be de-facto neglected. Everyone does what's "hot", so all the tutorials and bug reports and tests and SO questions and new libraries and and and all go towards that framework or language or tool or method. You can't even argue technical merits towards the neglected options because yes the popular tool is better, but only because we have a metric boat load (millions) of man-hours being pumped into making it better instead of all the alternatives. It's like the tech-equivalent of fashion fads in that it's self-reinforcing. Not to take away from some of the actual and technical achievements that some of these things have made, of course.
- u678u 5y ago
  You should see our k8s migration, its the biggest waste of time for everyone I've ever seen.
- jonnycomputer 5y ago
  I guess. But frankly, all the solutions were than not that great. So its not surprising people might want to move to something new.

KaiserPro 5y ago

Conda is a ginormous pain in the arse. I supported a research cluster, and the amount of times conda caused an issue was a lot.

pmkiwi 5y ago

conda makes sense in some circumstances but you need to invest a lot of times to undersand this tool (remind me my Maven times sometimes...). With all scientist packages, conda is pretty good but it is sooooo slow, it is just insane!