3 ms·

This is probably a hard problem, but the dependency graph resolution in conda is a thing of nightmares. It is so appallingly slow even for relatively menial tas

This is probably a hard problem, but the dependency graph resolution in conda is a thing of nightmares. It is so appallingly slow even for relatively menial tasks that updating environments becomes an exercise in frustration. I'm unsure what are the core issues, but in my experience cargo was always pretty quick to use and if it fails, it fails fast. Conda on the other hand is slow for the simple cases, and if the graph becomes complex it will just churn for 15 minutes before throwing it's hands in the air and giving up with some cryptic error. I suspect it comes back to the fact that packaging and dependency management was thought about upfront for Rust and the whole ecosystem was built well from the get go?

steveklabnik 6y ago
I've been meaning to write a blog post on this topic. That is, on the various speeds of various package managers. I don't know a ton about Conda, but when I was looking into Poetry, one of the core issues is: https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow https://python-poetry.org/docs/faq/#why-is-the-dependency-re... > This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process. Cargo doesn't need to do anything with packages directly to do its job; everything it needs is in the index. This makes it pretty fast.
- estebank 6y ago
  >> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process. This sounds like something that could be done server side, either by PyPI or another entity and expose it through a new API endpoint, instead of doing it on every Python developer's machine.
  - steveklabnik 6y ago
    The core issue is that setup.py can be non-deterministic, so them doing it server side may not give the right results. https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies/ https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-k...
stevesimmons 6y ago
conda taking 15+ mins to resolve dependencies is by far its biggest weakness.
- benhurmarcel 6y ago
  If it doesn't fail to "solve the environment" after 30 minutes that's already a win.

steveklabnik 6y ago

I've been meaning to write a blog post on this topic. That is, on the various speeds of various package managers. I don't know a ton about Conda, but when I was looking into Poetry, one of the core issues is: https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow https://python-poetry.org/docs/faq/#why-is-the-dependency-re... > This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process. Cargo doesn't need to do anything with packages directly to do its job; everything it needs is in the index. This makes it pretty fast.

estebank 6y ago
>> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process. This sounds like something that could be done server side, either by PyPI or another entity and expose it through a new API endpoint, instead of doing it on every Python developer's machine.
- steveklabnik 6y ago
  The core issue is that setup.py can be non-deterministic, so them doing it server side may not give the right results. https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies/ https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-k...

stevesimmons 6y ago

conda taking 15+ mins to resolve dependencies is by far its biggest weakness.

benhurmarcel 6y ago
If it doesn't fail to "solve the environment" after 30 minutes that's already a win.