3 ms·

I really hope conda stays on the user end. Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. Bu

I really hope conda stays on the user end. Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved. Not to mention conda installs seem to take waaaay longer than pip. apt + pip works far more sensibly and reliably in my experience.

blt 6y ago
conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups? [1] https://research.swtch.com/version-sat https://research.swtch.com/version-sat
- downerending 6y ago
  It can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right. Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
- wdobbels 6y ago
  Indeed, pip has no dependency resolver as of now, although it is being actively worked on [1], and they did secure some funding [2]. [1] https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 [2] https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html https://pyfound.blogspot.com/2019/11/seeking-developers-for-...
- dimatura 6y ago
  Yeah, so while conda can be annoyingly slow, comparing it directly to pip in that department is unfair. A more fair comparison would be pip plus poetry or pipenv, which also have speed complaints :/
lsorber 6y ago
My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.
- dijksterhuis 6y ago
  Yeah I know I'm on one side of the fence with this. The people I work with (not engineers) use conda all the time and love it. A big thing for me is that apt + pip means I know more about what I'm installing, whereas conda seems like it'll go off and do what it thinks is best. If it's going into production then I want to know why we need package X and how it's been installed, rather than "conda says we need it". Basically, I think conda encourages "install and forget" which means people don't really know or understand what they're putting into production. And that can cause a lot of problems further down the line. Also the fact that conda installs everything to sandbox causes it's own issues. Suddenly I have two versions of a system package. Now I've got to do extra work to deal with that. Then again, it could just be I never got over the time where uninstalling anaconda ripped apart the python install on my MacBook. That's a weekend I won't get back. __ Also, Production shouldn't use virtual environments in my opinion. That's an additional deployment/build step which could fail one day. The container image is a virtual environment in and of itself anyway!
  - akbo 6y ago
    Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-dependency-management/ https://haveagreatdata.com/posts/data-science-python-depende... Re the extra "virtual environment", you can just use conda's base environment in prod. Here's how to do that: https://haveagreatdata.com/posts/step-by-step-docker-image-for-data-science-projects/ https://haveagreatdata.com/posts/step-by-step-docker-image-f...
mbar84 6y ago
I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like. https://gitlab.com/mbarkhau/bootstrapit https://gitlab.com/mbarkhau/bootstrapit

blt 6y ago

conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups? [1] https://research.swtch.com/version-sat https://research.swtch.com/version-sat

downerending 6y ago
It can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right. Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
wdobbels 6y ago
Indeed, pip has no dependency resolver as of now, although it is being actively worked on [1], and they did secure some funding [2]. [1] https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 [2] https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html https://pyfound.blogspot.com/2019/11/seeking-developers-for-...
dimatura 6y ago
Yeah, so while conda can be annoyingly slow, comparing it directly to pip in that department is unfair. A more fair comparison would be pip plus poetry or pipenv, which also have speed complaints :/

lsorber 6y ago

My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.

dijksterhuis 6y ago
Yeah I know I'm on one side of the fence with this. The people I work with (not engineers) use conda all the time and love it. A big thing for me is that apt + pip means I know more about what I'm installing, whereas conda seems like it'll go off and do what it thinks is best. If it's going into production then I want to know why we need package X and how it's been installed, rather than "conda says we need it". Basically, I think conda encourages "install and forget" which means people don't really know or understand what they're putting into production. And that can cause a lot of problems further down the line. Also the fact that conda installs everything to sandbox causes it's own issues. Suddenly I have two versions of a system package. Now I've got to do extra work to deal with that. Then again, it could just be I never got over the time where uninstalling anaconda ripped apart the python install on my MacBook. That's a weekend I won't get back. __ Also, Production shouldn't use virtual environments in my opinion. That's an additional deployment/build step which could fail one day. The container image is a virtual environment in and of itself anyway!
- akbo 6y ago
  Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-dependency-management/ https://haveagreatdata.com/posts/data-science-python-depende... Re the extra "virtual environment", you can just use conda's base environment in prod. Here's how to do that: https://haveagreatdata.com/posts/step-by-step-docker-image-for-data-science-projects/ https://haveagreatdata.com/posts/step-by-step-docker-image-f...

mbar84 6y ago

I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like. https://gitlab.com/mbarkhau/bootstrapit https://gitlab.com/mbarkhau/bootstrapit