5 ms·

So far it's just a bit of smoke on the horizon, but I'm noticing some packages abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to tell

So far it's just a bit of smoke on the horizon, but I'm noticing some packages abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to tell if this trend will take off, but it does seem plausible.

dijksterhuis 6y ago
I really hope conda stays on the user end. Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved. Not to mention conda installs seem to take waaaay longer than pip. apt + pip works far more sensibly and reliably in my experience.
- blt 6y ago
  conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups? [1] https://research.swtch.com/version-sat https://research.swtch.com/version-sat
  - downerending 6y ago
    It can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right. Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
  - wdobbels 6y ago
    Indeed, pip has no dependency resolver as of now, although it is being actively worked on [1], and they did secure some funding [2]. [1] https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 [2] https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html https://pyfound.blogspot.com/2019/11/seeking-developers-for-...
  - dimatura 6y ago
    Yeah, so while conda can be annoyingly slow, comparing it directly to pip in that department is unfair. A more fair comparison would be pip plus poetry or pipenv, which also have speed complaints :/
- lsorber 6y ago
  My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.
  - dijksterhuis 6y ago
    Yeah I know I'm on one side of the fence with this. The people I work with (not engineers) use conda all the time and love it. A big thing for me is that apt + pip means I know more about what I'm installing, whereas conda seems like it'll go off and do what it thinks is best. If it's going into production then I want to know why we need package X and how it's been installed, rather than "conda says we need it". Basically, I think conda encourages "install and forget" which means people don't really know or understand what they're putting into production. And that can cause a lot of problems further down the line. Also the fact that conda installs everything to sandbox causes it's own issues. Suddenly I have two versions of a system package. Now I've got to do extra work to deal with that. Then again, it could just be I never got over the time where uninstalling anaconda ripped apart the python install on my MacBook. That's a weekend I won't get back. __ Also, Production shouldn't use virtual environments in my opinion. That's an additional deployment/build step which could fail one day. The container image is a virtual environment in and of itself anyway!
    - akbo 6y ago
      Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-dependency-management/ https://haveagreatdata.com/posts/data-science-python-depende... Re the extra "virtual environment", you can just use conda's base environment in prod. Here's how to do that: https://haveagreatdata.com/posts/step-by-step-docker-image-for-data-science-projects/ https://haveagreatdata.com/posts/step-by-step-docker-image-f...
- mbar84 6y ago
  I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like. https://gitlab.com/mbarkhau/bootstrapit https://gitlab.com/mbarkhau/bootstrapit
closeparen 6y ago
As I understand it, conda has more sophisticated handling of native components, and is largely seen in scientific environments where Python is just the frontend to libraries that are actually in C++ and Fortran.
0az 6y ago
Conda does their own thing, so I hope that this doesn't become commonplace. It's a nightmare of interop. Yes, it works for one person on a single laptop, but my experiences with conda outside of the happy path are universally terrible. I'll stick with the standards.
- lsorber 6y ago
  Can you give an example of a case outside of the happy path?
  - reubenmorais 6y ago
    Just one from my personal experience: Conda installed Python having different compilation flags than official and distro Python builds, breaking ABI and causing native extensions built against official Python to crash when loaded in Conda Python.
  - kaybe 6y ago
    A few times a year our office loses power, which causes the machine we have running for data evaluation to crash. There seems to be about a third chance for that to damage my Anaconda installation. Last time I could only solve it by deinstalling everything and reinstalling an updated version. Took ages to get all packages running again. Luckily my code still works but now I get depreciation warnings (and not in my code). Who knows what will happen next time. It only took like a day I needed for actually working with the data as well... urgh.
pas 6y ago
Conda is "pip" for people who know _nothing_ about pip/operating-systems/IT. It is basically a way to have a working environment without understanding that maybe pandas and numpy and scikit are not Python built-in modules, and that maybe you'd need a working C/C++ (and even a Fortran compiler too) to install it. It's great for that purpose, but - as far as I know - it has not much to do with general Python dependency management.
- downerending 6y ago
  Conda does (in theory) handle python module dependencies well. In practice, like pip, sometimes you win and sometimes you lose. Note that you can use 'pip' within a 'conda' environment as well, so you can sort of have it both ways.
dimatura 6y ago
I think this trend may be limited to fields relying heavily on natively compiled dependencies, which may not even have any python at all. Conda packages can be C++, Fortran, pretty much anything. That's the main draw for me -- otherwise, I'd stick with pip, which is definitely more mainstream.

dijksterhuis 6y ago

I really hope conda stays on the user end. Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved. Not to mention conda installs seem to take waaaay longer than pip. apt + pip works far more sensibly and reliably in my experience.

blt 6y ago
conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups? [1] https://research.swtch.com/version-sat https://research.swtch.com/version-sat
- downerending 6y ago
  It can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right. Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
- wdobbels 6y ago
  Indeed, pip has no dependency resolver as of now, although it is being actively worked on [1], and they did secure some funding [2]. [1] https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 [2] https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html https://pyfound.blogspot.com/2019/11/seeking-developers-for-...
- dimatura 6y ago
  Yeah, so while conda can be annoyingly slow, comparing it directly to pip in that department is unfair. A more fair comparison would be pip plus poetry or pipenv, which also have speed complaints :/
lsorber 6y ago
My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.
- dijksterhuis 6y ago
  Yeah I know I'm on one side of the fence with this. The people I work with (not engineers) use conda all the time and love it. A big thing for me is that apt + pip means I know more about what I'm installing, whereas conda seems like it'll go off and do what it thinks is best. If it's going into production then I want to know why we need package X and how it's been installed, rather than "conda says we need it". Basically, I think conda encourages "install and forget" which means people don't really know or understand what they're putting into production. And that can cause a lot of problems further down the line. Also the fact that conda installs everything to sandbox causes it's own issues. Suddenly I have two versions of a system package. Now I've got to do extra work to deal with that. Then again, it could just be I never got over the time where uninstalling anaconda ripped apart the python install on my MacBook. That's a weekend I won't get back. __ Also, Production shouldn't use virtual environments in my opinion. That's an additional deployment/build step which could fail one day. The container image is a virtual environment in and of itself anyway!
  - akbo 6y ago
    Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-dependency-management/ https://haveagreatdata.com/posts/data-science-python-depende... Re the extra "virtual environment", you can just use conda's base environment in prod. Here's how to do that: https://haveagreatdata.com/posts/step-by-step-docker-image-for-data-science-projects/ https://haveagreatdata.com/posts/step-by-step-docker-image-f...
mbar84 6y ago
I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like. https://gitlab.com/mbarkhau/bootstrapit https://gitlab.com/mbarkhau/bootstrapit

closeparen 6y ago

As I understand it, conda has more sophisticated handling of native components, and is largely seen in scientific environments where Python is just the frontend to libraries that are actually in C++ and Fortran.

0az 6y ago

Conda does their own thing, so I hope that this doesn't become commonplace. It's a nightmare of interop. Yes, it works for one person on a single laptop, but my experiences with conda outside of the happy path are universally terrible. I'll stick with the standards.

lsorber 6y ago
Can you give an example of a case outside of the happy path?
- reubenmorais 6y ago
  Just one from my personal experience: Conda installed Python having different compilation flags than official and distro Python builds, breaking ABI and causing native extensions built against official Python to crash when loaded in Conda Python.
- kaybe 6y ago
  A few times a year our office loses power, which causes the machine we have running for data evaluation to crash. There seems to be about a third chance for that to damage my Anaconda installation. Last time I could only solve it by deinstalling everything and reinstalling an updated version. Took ages to get all packages running again. Luckily my code still works but now I get depreciation warnings (and not in my code). Who knows what will happen next time. It only took like a day I needed for actually working with the data as well... urgh.

pas 6y ago

Conda is "pip" for people who know _nothing_ about pip/operating-systems/IT. It is basically a way to have a working environment without understanding that maybe pandas and numpy and scikit are not Python built-in modules, and that maybe you'd need a working C/C++ (and even a Fortran compiler too) to install it. It's great for that purpose, but - as far as I know - it has not much to do with general Python dependency management.

downerending 6y ago
Conda does (in theory) handle python module dependencies well. In practice, like pip, sometimes you win and sometimes you lose. Note that you can use 'pip' within a 'conda' environment as well, so you can sort of have it both ways.

dimatura 6y ago

I think this trend may be limited to fields relying heavily on natively compiled dependencies, which may not even have any python at all. Conda packages can be C++, Fortran, pretty much anything. That's the main draw for me -- otherwise, I'd stick with pip, which is definitely more mainstream.