3 ms·

I wish there is some package manager in middle of conda and pip. Conda is too strict and often get stuck in SAT solving. pip doesn't even ask when reinstalling

I wish there is some package manager in middle of conda and pip. Conda is too strict and often get stuck in SAT solving. pip doesn't even ask when reinstalling a version currently being used. Edit: Typo: reinstalling a version of package currently being used

woodruffw 4y ago
> pip doesn't even ask when reinstalling a version currently being used. Just as an explanation: a "version" in Python packaging can come from one of many potential distributions, including a local distribution (such as a path on disk) that might different from a canonical released distribution on PyPI. Having `pip install ...` always re-install based on its candidate selection rules is generally good (IMO), since an explicit run of `pip install` implies user intent to search for a potentially new or changed distribution.
- YetAnotherNick 4y ago
  I meant this for dependency not the package I am installing.
  - woodruffw 4y ago
    I'm not sure I understand what you mean -- `pip` should not be reinstalling transitive dependencies. If you install A and B and both depend on C, C should only be installed once.
    - Izkata 4y ago
      I think they're referring to requirements files. I've seen the same behavior - on day 1, pip installs packages A and B, then on day 2 when someone else modified the requirements file it installs C and reinstalls B even though B hasn't changed. This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade". There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.
lmeyerov 4y ago
we find mamba solves deps solving for conda (we fail to do GPU dependencies without it), and I think it's getting integrated my main thing w/ conda is it's bananas figuring out how to make a new recipe, which is pretty surprising
- kylebarron 4y ago
  Agreed, I've found packaging for conda to be so much harder than packaging for pip
thefinaluser 4y ago
Try poetry. It wraps pip and fixes a lot of its issues
- lightspot21 4y ago
  Seconding Poetry. IMO it should have been the standard package manager - it just works (TM)
  - Smaug123 4y ago
    Although it consumed 70GB of RAM before I killed it, when I tried to use it to `poetry install` stable-diffusion.
dwagnerkc 4y ago
Try using mamba (https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba) We ran into many unsolvable or 30m+ solvable envs with conda that mamba handled quickly. The underlying solver can be used with conda directly as well, but I have not done that (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...)
5d8767c68926 4y ago
My ask would be to get rid of the need for conda all together. Conda obviously offers a lot of value in sharing hairy compiled packages, but it does not play well with anything else. None of available the tooling really works with both conda and pip. It fragments the already lousy packaging story.
- insane_dreamer 4y ago
  other than an occasional pkg conflict, we've found conda and pip to work pretty well together (defining pip dependencies in the conda env) -- across a lot of different python envs
smeagull 4y ago
Conda sucks because it always wants things its own way, and introduces yet another site-packages and environment I need to track. Absolutely hate Apple for requiring it for their metal and M1 support.
dec0dedab0de 4y ago
Sounds like you want pipenv, but that might be too close to conda

woodruffw 4y ago

> pip doesn't even ask when reinstalling a version currently being used. Just as an explanation: a "version" in Python packaging can come from one of many potential distributions, including a local distribution (such as a path on disk) that might different from a canonical released distribution on PyPI. Having `pip install ...` always re-install based on its candidate selection rules is generally good (IMO), since an explicit run of `pip install` implies user intent to search for a potentially new or changed distribution.

YetAnotherNick 4y ago
I meant this for dependency not the package I am installing.
- woodruffw 4y ago
  I'm not sure I understand what you mean -- `pip` should not be reinstalling transitive dependencies. If you install A and B and both depend on C, C should only be installed once.
  - Izkata 4y ago
    I think they're referring to requirements files. I've seen the same behavior - on day 1, pip installs packages A and B, then on day 2 when someone else modified the requirements file it installs C and reinstalls B even though B hasn't changed. This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade". There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.

lmeyerov 4y ago

we find mamba solves deps solving for conda (we fail to do GPU dependencies without it), and I think it's getting integrated my main thing w/ conda is it's bananas figuring out how to make a new recipe, which is pretty surprising

kylebarron 4y ago
Agreed, I've found packaging for conda to be so much harder than packaging for pip

thefinaluser 4y ago

Try poetry. It wraps pip and fixes a lot of its issues

lightspot21 4y ago
Seconding Poetry. IMO it should have been the standard package manager - it just works (TM)
- Smaug123 4y ago
  Although it consumed 70GB of RAM before I killed it, when I tried to use it to `poetry install` stable-diffusion.

dwagnerkc 4y ago

Try using mamba (https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba) We ran into many unsolvable or 30m+ solvable envs with conda that mamba handled quickly. The underlying solver can be used with conda directly as well, but I have not done that (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...)

5d8767c68926 4y ago

My ask would be to get rid of the need for conda all together. Conda obviously offers a lot of value in sharing hairy compiled packages, but it does not play well with anything else. None of available the tooling really works with both conda and pip. It fragments the already lousy packaging story.

insane_dreamer 4y ago
other than an occasional pkg conflict, we've found conda and pip to work pretty well together (defining pip dependencies in the conda env) -- across a lot of different python envs

smeagull 4y ago

Conda sucks because it always wants things its own way, and introduces yet another site-packages and environment I need to track. Absolutely hate Apple for requiring it for their metal and M1 support.

dec0dedab0de 4y ago

Sounds like you want pipenv, but that might be too close to conda