3 ms·

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people nee

by zzzeek 4y ago

it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nice but not something I've ever needed, but the buzz got out and it took over, people who have never typed "virtualenv" use poetry and they have no idea why.

teruakohatu 4y ago
> vague impression is that scence-y types Because often python is only one part of the puzzle, we also need a variety of third party libraries and maybe R as well (along with R packages). And we would rather not have to explain to another researcher how to install it all.
squeaky-clean 4y ago
Conda is very nice for people who just want to write some python and not need to learn about the ecosystem or weird issues like some Visual Studio dependency not being installed to get scipy to compile on Windows or whatever. Like someone coming from R to Python. But aside from that situation I agree with you.
pletnes 4y ago
Before conda, you needed a C and a fortran compiler, a BLAS and LAPACK installation, and various build tools, to install scipy. Scipy is one - 1 - dependency used in science and engineering. The pain used to be massive. Now, we’re complaining about the existence of 2 high quality (not perfect) alternatives, and pip (part of the official python distro) can install tensorflow etc, GPU libs included, with a oneliner.
- Doxin 4y ago
  These days with wheels there is 0 reason why there can't be binary pip packages for any of those. Definitely used to be a problem though.
jayvius 4y ago
> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). Essentially, yes, and justifiably so. Try installing science-y Python packages on Windows written in C. When conda was created in 2012 this meant installing Visual Studio 2005 (for Python 2.7) which was hard to find on Microsoft's own website even back then.
- bombolo 4y ago
  > on Windows Ah, I found your bug!
zzzeek 4y ago
so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2. the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ https://pypi.org/project/scipy/ with a few dozen distros. is the problem solved yet ?
- Filligree 4y ago
  You can't install the CUDA base libraries with pip. You can with Conda. So no, it isn't solved yet.
  - buildbot 4y ago
    Good? I don't want my package manager messing with my cuda toolkit setup!
    - tryauuum 4y ago
      ? what if you need two different versions of cuda toolkit on same machine?
      - buildbot 4y ago
        I may be wrong, but I don’t think you would want that - it should match with your driver version. Weird stuff happens with minor version mismatches
        Filligree 4y ago
        Probably true, but it usually works anyway. And a lot of ML code depends on exact versions of the CUDA libraries, so you don't get a choice.
crabbone 4y ago
> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). It started in a way similar to how Active Perl started. An attempt to build a reliable, curated and "batteries included" solution. (But it went downhill very quickly). So, what were the problems: * Most importantly, using Python on Windows is a pain (it still is, but less so). Many Python packages rely on native libraries which they don't provide. Even if they ship with some binary blob, that blob is often just a wrapper to the actual library. But, at the time Conda was created, even a binary blob was not an option. If you wanted to install stuff, you had to compile it. And, compiling on MS Windows is a world of pain. Especially, if you compile something for Python. Again, credit where it's due, Python sort of fixed some of these problems... but just a tiny bit. It still sucks a lot. So, conda is more of a MSYS2 really: it's a Linux user-space in Windows (firstly), and Python on top of that. But, it kind of also doesn't want to admit it, so, it still pretends it's Windows... but with a bunch of Linux-ey stuff... * Secondly, pip was a joke until about a year ago. It was not really a package manager (it still isn't) and it couldn't properly solve dependencies (it sort of does today, but poorly because of source-distributions). Conda, when it worked, was at least somewhat consistent. (But usually, it just hangs). Also, conda is a package manager (however awful). I.e. with pip you can run install once, and then you can run a different install command again in the same environment, and god help you to make sure you have consistent dependencies between packages. Conda, in principle, should always keep your environment consistent. (But it hangs more often than not). Also, there's no such thing as installing from source with conda. If you want to use source: git clone, install conda-build and suffer through non-existent documentation and get help from non-existent community of people building conda packages. * Conda provides a curated set of packages (a.k.a default channel). Anyone worth their salt ditches that channel the moment they install conda and installs everything from conda-forge (cuz its got more stuffs!) So, they tried curated. Didn't work. --- In a sense, I think that what happened is that conda was too ambitious of a project, using wrong technology to implement it. It also didn't have good enough planning for the kind of ambition they had. So, every time they came to solve a real problem, they didn't have time, human resources and system resources to solve it well. They've accumulated technical debt at close-to-light speed and ended up being a huge mess nobody knows how to deal with. Some of the obvious mistakes would be: creating too many releases of conflicting packages. They should've worked on some sort of LTS solution, where they release a set of packages with very permissive dependencies of very few versions that had been proven to work well together. Instead it's very common for conda packages to be very peculiar (and without real need to do so) about the versions of their dependencies. Conda people didn't build good CI. They often release absolutely broken packages, and only discover it retrospectively from community input. (Tensorflow would be a great example). This creates a lot of problems with automatic updates. Conda gave in to community pressure and decided to build integration with pip. It doesn't work well, and it's not possible to make it work well, but they added this and a lot of people instantly created dependencies on this feature. Conda picked a bone with some of the infra tools outside of Python ecosystem. In particular with CMake. This just adds an extra aspect of pain trying to build conda packages / work with native libraries / wrappers. It might seem like a minor thing, but it prevented a lot of people who might otherwise release packages both for conda and PyPI from doing so. Often package that is released to conda is ported by someone who's not the package author. It also means that sometimes the names of packages differ between conda and PyPI for the same package. ---- NB. In terms of amount of commands you need to type when working with conda vs with PyPI tools is not noticeably different. Conda is, perhaps, more organized, but is also more buggy due to trying to integrate with various shells in special ways, and failing quite a bit.
int_19h 4y ago
conda exists because it can install so much more stuff than pip without hassle, including things that aren't themselves Python packages, but which Python packages need. For example, if you ever tried to build a Python module that has some native code on Windows, you might appreciate this: https://anaconda.org/conda-forge/cxx-compiler https://anaconda.org/conda-forge/cxx-compiler.

by zzzeek 4y ago

teruakohatu 4y ago

> vague impression is that scence-y types Because often python is only one part of the puzzle, we also need a variety of third party libraries and maybe R as well (along with R packages). And we would rather not have to explain to another researcher how to install it all.

squeaky-clean 4y ago

Conda is very nice for people who just want to write some python and not need to learn about the ecosystem or weird issues like some Visual Studio dependency not being installed to get scipy to compile on Windows or whatever. Like someone coming from R to Python. But aside from that situation I agree with you.

pletnes 4y ago

Before conda, you needed a C and a fortran compiler, a BLAS and LAPACK installation, and various build tools, to install scipy. Scipy is one - 1 - dependency used in science and engineering. The pain used to be massive. Now, we’re complaining about the existence of 2 high quality (not perfect) alternatives, and pip (part of the official python distro) can install tensorflow etc, GPU libs included, with a oneliner.

Doxin 4y ago
These days with wheels there is 0 reason why there can't be binary pip packages for any of those. Definitely used to be a problem though.

jayvius 4y ago

> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). Essentially, yes, and justifiably so. Try installing science-y Python packages on Windows written in C. When conda was created in 2012 this meant installing Visual Studio 2005 (for Python 2.7) which was hard to find on Microsoft's own website even back then.

bombolo 4y ago
> on Windows Ah, I found your bug!

zzzeek 4y ago

so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2. the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ https://pypi.org/project/scipy/ with a few dozen distros. is the problem solved yet ?

Filligree 4y ago
You can't install the CUDA base libraries with pip. You can with Conda. So no, it isn't solved yet.
- buildbot 4y ago
  Good? I don't want my package manager messing with my cuda toolkit setup!
  - tryauuum 4y ago
    ? what if you need two different versions of cuda toolkit on same machine?
    - buildbot 4y ago
      I may be wrong, but I don’t think you would want that - it should match with your driver version. Weird stuff happens with minor version mismatches
      - Filligree 4y ago
        Probably true, but it usually works anyway. And a lot of ML code depends on exact versions of the CUDA libraries, so you don't get a choice.

crabbone 4y ago

> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). It started in a way similar to how Active Perl started. An attempt to build a reliable, curated and "batteries included" solution. (But it went downhill very quickly). So, what were the problems: * Most importantly, using Python on Windows is a pain (it still is, but less so). Many Python packages rely on native libraries which they don't provide. Even if they ship with some binary blob, that blob is often just a wrapper to the actual library. But, at the time Conda was created, even a binary blob was not an option. If you wanted to install stuff, you had to compile it. And, compiling on MS Windows is a world of pain. Especially, if you compile something for Python. Again, credit where it's due, Python sort of fixed some of these problems... but just a tiny bit. It still sucks a lot. So, conda is more of a MSYS2 really: it's a Linux user-space in Windows (firstly), and Python on top of that. But, it kind of also doesn't want to admit it, so, it still pretends it's Windows... but with a bunch of Linux-ey stuff... * Secondly, pip was a joke until about a year ago. It was not really a package manager (it still isn't) and it couldn't properly solve dependencies (it sort of does today, but poorly because of source-distributions). Conda, when it worked, was at least somewhat consistent. (But usually, it just hangs). Also, conda is a package manager (however awful). I.e. with pip you can run install once, and then you can run a different install command again in the same environment, and god help you to make sure you have consistent dependencies between packages. Conda, in principle, should always keep your environment consistent. (But it hangs more often than not). Also, there's no such thing as installing from source with conda. If you want to use source: git clone, install conda-build and suffer through non-existent documentation and get help from non-existent community of people building conda packages. * Conda provides a curated set of packages (a.k.a default channel). Anyone worth their salt ditches that channel the moment they install conda and installs everything from conda-forge (cuz its got more stuffs!) So, they tried curated. Didn't work. --- In a sense, I think that what happened is that conda was too ambitious of a project, using wrong technology to implement it. It also didn't have good enough planning for the kind of ambition they had. So, every time they came to solve a real problem, they didn't have time, human resources and system resources to solve it well. They've accumulated technical debt at close-to-light speed and ended up being a huge mess nobody knows how to deal with. Some of the obvious mistakes would be: creating too many releases of conflicting packages. They should've worked on some sort of LTS solution, where they release a set of packages with very permissive dependencies of very few versions that had been proven to work well together. Instead it's very common for conda packages to be very peculiar (and without real need to do so) about the versions of their dependencies. Conda people didn't build good CI. They often release absolutely broken packages, and only discover it retrospectively from community input. (Tensorflow would be a great example). This creates a lot of problems with automatic updates. Conda gave in to community pressure and decided to build integration with pip. It doesn't work well, and it's not possible to make it work well, but they added this and a lot of people instantly created dependencies on this feature. Conda picked a bone with some of the infra tools outside of Python ecosystem. In particular with CMake. This just adds an extra aspect of pain trying to build conda packages / work with native libraries / wrappers. It might seem like a minor thing, but it prevented a lot of people who might otherwise release packages both for conda and PyPI from doing so. Often package that is released to conda is ported by someone who's not the package author. It also means that sometimes the names of packages differ between conda and PyPI for the same package. ---- NB. In terms of amount of commands you need to type when working with conda vs with PyPI tools is not noticeably different. Conda is, perhaps, more organized, but is also more buggy due to trying to integrate with various shells in special ways, and failing quite a bit.

int_19h 4y ago

conda exists because it can install so much more stuff than pip without hassle, including things that aren't themselves Python packages, but which Python packages need. For example, if you ever tried to build a Python module that has some native code on Windows, you might appreciate this: https://anaconda.org/conda-forge/cxx-compiler https://anaconda.org/conda-forge/cxx-compiler.