3 ms·

Oh, this is about Conda?.. I think, this is going to be the third go at the problem? The first two were hands-down awful leading to creation of Mamba. But sp

Oh, this is about Conda?.. I think, this is going to be the third go at the problem? The first two were hands-down awful leading to creation of Mamba. But speeding things up is only part of the solution here. Conda package ecosystem has a bunch of issues that are related to the format and the repositories providing packages as well as on the policy for accepting packages into those repositories. They tried to solve too many problems at once, probably, w/o even realizing what kind of problem they are attacking. Here's a major problem with Conda ecosystem, the way I see it: package developers are encouraged to produce very "precise" dependency requirements because this makes installation process go faster and ensures less wiggle room for untested dependency combination. This, in turn, results in proliferation of package versions, all of which declare themselves as incompatible and non-interchangeable with many others. On top of this, beside providing just the Python packages, Conda, ambitiously, wants to also provide shared libraries, which blows up the minimum installation to an enormous size, including the number of packages that need installing. So, being a Conda package maintainer, you will face a dilemma: do you want your users to be able to install the package in acceptable time (at least under 10 minutes...) or work towards compatibility with multiple different versions of other packages the users might need for other reasons in the same environment (and your installation times start shooting through the roof). In my experience, the longest successful Conda install took about a day (i.e. close to 24 hours). It may take few days for Conda to fail to install (due to imaginary or real conflict). Now, add to this the existence of multiple sources for different packages (i.e. "channels"). Different channels may, and actually quite often do provide different unrelated other packages in their channel, but the users cannot specify the channel when requesting to install a package on per-package basis. So, the solver needs to guess somehow which channel to prefer as a source of any given package. The problem is that while the package version found in that channel might be the same, the requirements for that package might differ between channels. So, while aiming at providing a lot of possibly useful features, Conda programmers programmed themselves into a corner with no good way out. Several things might help, but none will ultimately solve the situation. * Package bundles (or meta-packages, like they are called in some Linux distributions) will cut significantly on the work a solver needs to do. A lot of the basic (shared libraries) packages are, essentially, always the same in each install. * LTS curated channel snapshots. I.e. provide a very large collection of packages that are verified to all work together, but unlike the meta-packages need not be installed together. * Incorporate channel into version specification, especially when specifying package dependencies so that maintainers were required to specify the source of their dependencies when submitting their package. * There are a bunch of (very) infrequently used options in the current solver that can be safely removed, but they add to the complexity and the time the solver spends in various "impossible" scenarios.

droelf 3y ago
Let me try to reply to you (as one of the authors of mamba and as a conda-forge core dev): Mamba has solved (seemingly successfuly) many problems of conda being slow. It's used by default in conda-forge, the largest conda repository out there. Asking packagers to add upper bounds is less of a speed-hack vs. just the correct thing to do. What's interesting about conda-forge is that upper bounds can also be added later on via a "repodata patch". They really just serve to get users packages that are actually compatible. The kind of "freak solves" are not a thing anymore with mamba. I agree with you that multiple channels present problems. Channels that explicitly inherit from each other work quite well together (e.g. bioconda and robostack channels extend conda-forge), but the Anaconda main (defaults) channel and conda-forge do not work well together – on the metadata and ABI level. For this reason we encourage users to never mix those two. At prefix.dev we do want to make it easy to build "on top of" conda-forge in the future. I think most issues on your list that follows at the end are non-issues once you start to use mamba. And I would encourage you wholeheartedly to give micromamba a try (for a really fast, single-binary experience with no base env and slim installation) or pixi (again, no base env, just a Rust binary).
- crabbone 3y ago
  I don't really use conda or mamba for my own things. My interaction with these is in supporting others who need to use a package that I help develop. So, even if mamba works better (as in faster), it doesn't help me, as my goal is to make sure the package installs with what the users will have. Knowing my audience, it's hard to get them to install anything, no matter how beneficial to them. Also, to their defense, any additional step they need to make when installing is an extra failure point, or at least adds more labor and confusion to the process most users see as tedious and uninteresting -- setting their environment. So, unless mamba replaces conda, I still need to support conda. Anyways. The biggest problem isn't even the speed. It's the conceptual problem. Both mamba and conda are pushing package maintainers towards very (unnecessarily) precise dependency version specification. This creates a lot of interoperability problems. Software goes "stale" very quickly. Since this is often used in scientific setting, the research reproducibility suffers a lot from this. The blame is only partially with mamba / conda. The other bad thing that is happening in this environment is the lack of standards. Even if dependency on anything below minor version was prohibited, it would slow down the software rot, but not by much. Minor Python versions go stale after about four iterations, which is something like four years. Meaning that software that was written five years ago, if it wasn't maintained during that time is in most likelihood unusable today. Yet another problem is the lack of organization in this environment. Enterprise Linux distributions are trying to ensure relative stability of the system they provide to the customer by testing third-party packages provided with the system together with the system. Nobody does anything comparable to it in Python world. This results in situations where installation performed from the same requirements days apart installs different packages (and of course you notice it when things start to break as a result). Even worse, when things don't immediately break, but the results of the research start to differ day on day. In my view, Python is a bad choice for research due to how its environment is organized (mostly not organized). But, I guess, that the current policy is to sacrifice any desirable engineering qualities one might want from an environment to popularity because that allows to onboard more researches who would otherwise be stuck with MS Excel and / or Matlab. It's sad though that there needs to be a compromise at all...

droelf 3y ago

Let me try to reply to you (as one of the authors of mamba and as a conda-forge core dev): Mamba has solved (seemingly successfuly) many problems of conda being slow. It's used by default in conda-forge, the largest conda repository out there. Asking packagers to add upper bounds is less of a speed-hack vs. just the correct thing to do. What's interesting about conda-forge is that upper bounds can also be added later on via a "repodata patch". They really just serve to get users packages that are actually compatible. The kind of "freak solves" are not a thing anymore with mamba. I agree with you that multiple channels present problems. Channels that explicitly inherit from each other work quite well together (e.g. bioconda and robostack channels extend conda-forge), but the Anaconda main (defaults) channel and conda-forge do not work well together – on the metadata and ABI level. For this reason we encourage users to never mix those two. At prefix.dev we do want to make it easy to build "on top of" conda-forge in the future. I think most issues on your list that follows at the end are non-issues once you start to use mamba. And I would encourage you wholeheartedly to give micromamba a try (for a really fast, single-binary experience with no base env and slim installation) or pixi (again, no base env, just a Rust binary).

crabbone 3y ago
I don't really use conda or mamba for my own things. My interaction with these is in supporting others who need to use a package that I help develop. So, even if mamba works better (as in faster), it doesn't help me, as my goal is to make sure the package installs with what the users will have. Knowing my audience, it's hard to get them to install anything, no matter how beneficial to them. Also, to their defense, any additional step they need to make when installing is an extra failure point, or at least adds more labor and confusion to the process most users see as tedious and uninteresting -- setting their environment. So, unless mamba replaces conda, I still need to support conda. Anyways. The biggest problem isn't even the speed. It's the conceptual problem. Both mamba and conda are pushing package maintainers towards very (unnecessarily) precise dependency version specification. This creates a lot of interoperability problems. Software goes "stale" very quickly. Since this is often used in scientific setting, the research reproducibility suffers a lot from this. The blame is only partially with mamba / conda. The other bad thing that is happening in this environment is the lack of standards. Even if dependency on anything below minor version was prohibited, it would slow down the software rot, but not by much. Minor Python versions go stale after about four iterations, which is something like four years. Meaning that software that was written five years ago, if it wasn't maintained during that time is in most likelihood unusable today. Yet another problem is the lack of organization in this environment. Enterprise Linux distributions are trying to ensure relative stability of the system they provide to the customer by testing third-party packages provided with the system together with the system. Nobody does anything comparable to it in Python world. This results in situations where installation performed from the same requirements days apart installs different packages (and of course you notice it when things start to break as a result). Even worse, when things don't immediately break, but the results of the research start to differ day on day. In my view, Python is a bad choice for research due to how its environment is organized (mostly not organized). But, I guess, that the current policy is to sacrifice any desirable engineering qualities one might want from an environment to popularity because that allows to onboard more researches who would otherwise be stuck with MS Excel and / or Matlab. It's sad though that there needs to be a compromise at all...