4 ms·

Let me try to reply to you (as one of the authors of mamba and as a conda-forge core dev): Mamba has solved (seemingly successfuly) many problems of conda bein

by droelf 3y ago

Let me try to reply to you (as one of the authors of mamba and as a conda-forge core dev): Mamba has solved (seemingly successfuly) many problems of conda being slow. It's used by default in conda-forge, the largest conda repository out there. Asking packagers to add upper bounds is less of a speed-hack vs. just the correct thing to do. What's interesting about conda-forge is that upper bounds can also be added later on via a "repodata patch". They really just serve to get users packages that are actually compatible. The kind of "freak solves" are not a thing anymore with mamba. I agree with you that multiple channels present problems. Channels that explicitly inherit from each other work quite well together (e.g. bioconda and robostack channels extend conda-forge), but the Anaconda main (defaults) channel and conda-forge do not work well together – on the metadata and ABI level. For this reason we encourage users to never mix those two. At prefix.dev we do want to make it easy to build "on top of" conda-forge in the future. I think most issues on your list that follows at the end are non-issues once you start to use mamba. And I would encourage you wholeheartedly to give micromamba a try (for a really fast, single-binary experience with no base env and slim installation) or pixi (again, no base env, just a Rust binary).

crabbone 3y ago
I don't really use conda or mamba for my own things. My interaction with these is in supporting others who need to use a package that I help develop. So, even if mamba works better (as in faster), it doesn't help me, as my goal is to make sure the package installs with what the users will have. Knowing my audience, it's hard to get them to install anything, no matter how beneficial to them. Also, to their defense, any additional step they need to make when installing is an extra failure point, or at least adds more labor and confusion to the process most users see as tedious and uninteresting -- setting their environment. So, unless mamba replaces conda, I still need to support conda. Anyways. The biggest problem isn't even the speed. It's the conceptual problem. Both mamba and conda are pushing package maintainers towards very (unnecessarily) precise dependency version specification. This creates a lot of interoperability problems. Software goes "stale" very quickly. Since this is often used in scientific setting, the research reproducibility suffers a lot from this. The blame is only partially with mamba / conda. The other bad thing that is happening in this environment is the lack of standards. Even if dependency on anything below minor version was prohibited, it would slow down the software rot, but not by much. Minor Python versions go stale after about four iterations, which is something like four years. Meaning that software that was written five years ago, if it wasn't maintained during that time is in most likelihood unusable today. Yet another problem is the lack of organization in this environment. Enterprise Linux distributions are trying to ensure relative stability of the system they provide to the customer by testing third-party packages provided with the system together with the system. Nobody does anything comparable to it in Python world. This results in situations where installation performed from the same requirements days apart installs different packages (and of course you notice it when things start to break as a result). Even worse, when things don't immediately break, but the results of the research start to differ day on day. In my view, Python is a bad choice for research due to how its environment is organized (mostly not organized). But, I guess, that the current policy is to sacrifice any desirable engineering qualities one might want from an environment to popularity because that allows to onboard more researches who would otherwise be stuck with MS Excel and / or Matlab. It's sad though that there needs to be a compromise at all...
code_biologist 3y ago
Asking packagers to add upper bounds is less of a speed-hack vs. just the correct thing to do. What's interesting about conda-forge is that upper bounds can also be added later on via a "repodata patch". I'm glad upper bounds can be updated, that definitely seems like the right approach. As a challenge on "just the correct thing to do", how should someone releasing a package using pandas specify an upper bound? Many basic dataframe uses will work just fine with pandas 3, whenever that happens. Some dataframe uses will break even in future minor releases. It's not clear to me that packagers can reliably predict that future. Another example of upper bounds causing pain: I mostly use poetry which unfortunately makes overrides extremely hard. A bunch of small Django packages work fine in Django 4 but haven't been updated in years and specify they only work for Django 3.x releases. The poetry folks say it's a packager problem, the packagers never respond to a PR, and off I go to make an internal fork of yet another library.
colechristensen 3y ago
Thank you for creating mamba! It took me from “I can’t install packages at all” to only occasional problems which is a huge win.

by droelf 3y ago

crabbone 3y ago

I don't really use conda or mamba for my own things. My interaction with these is in supporting others who need to use a package that I help develop. So, even if mamba works better (as in faster), it doesn't help me, as my goal is to make sure the package installs with what the users will have. Knowing my audience, it's hard to get them to install anything, no matter how beneficial to them. Also, to their defense, any additional step they need to make when installing is an extra failure point, or at least adds more labor and confusion to the process most users see as tedious and uninteresting -- setting their environment. So, unless mamba replaces conda, I still need to support conda. Anyways. The biggest problem isn't even the speed. It's the conceptual problem. Both mamba and conda are pushing package maintainers towards very (unnecessarily) precise dependency version specification. This creates a lot of interoperability problems. Software goes "stale" very quickly. Since this is often used in scientific setting, the research reproducibility suffers a lot from this. The blame is only partially with mamba / conda. The other bad thing that is happening in this environment is the lack of standards. Even if dependency on anything below minor version was prohibited, it would slow down the software rot, but not by much. Minor Python versions go stale after about four iterations, which is something like four years. Meaning that software that was written five years ago, if it wasn't maintained during that time is in most likelihood unusable today. Yet another problem is the lack of organization in this environment. Enterprise Linux distributions are trying to ensure relative stability of the system they provide to the customer by testing third-party packages provided with the system together with the system. Nobody does anything comparable to it in Python world. This results in situations where installation performed from the same requirements days apart installs different packages (and of course you notice it when things start to break as a result). Even worse, when things don't immediately break, but the results of the research start to differ day on day. In my view, Python is a bad choice for research due to how its environment is organized (mostly not organized). But, I guess, that the current policy is to sacrifice any desirable engineering qualities one might want from an environment to popularity because that allows to onboard more researches who would otherwise be stuck with MS Excel and / or Matlab. It's sad though that there needs to be a compromise at all...

code_biologist 3y ago

Asking packagers to add upper bounds is less of a speed-hack vs. just the correct thing to do. What's interesting about conda-forge is that upper bounds can also be added later on via a "repodata patch". I'm glad upper bounds can be updated, that definitely seems like the right approach. As a challenge on "just the correct thing to do", how should someone releasing a package using pandas specify an upper bound? Many basic dataframe uses will work just fine with pandas 3, whenever that happens. Some dataframe uses will break even in future minor releases. It's not clear to me that packagers can reliably predict that future. Another example of upper bounds causing pain: I mostly use poetry which unfortunately makes overrides extremely hard. A bunch of small Django packages work fine in Django 4 but haven't been updated in years and specify they only work for Django 3.x releases. The poetry folks say it's a packager problem, the packagers never respond to a PR, and off I go to make an internal fork of yet another library.

colechristensen 3y ago

Thank you for creating mamba! It took me from “I can’t install packages at all” to only occasional problems which is a huge win.