3 ms·

Cargo does semantic versioning-based dependency resolution, similar to Poetry, it's just a lot faster because it's written in a Rust. I don't think there's a bi

Cargo does semantic versioning-based dependency resolution, similar to Poetry, it's just a lot faster because it's written in a Rust. I don't think there's a big difference in the work that happens.

kissgyorgy 4y ago
I don't know anything about Rust, but probably there is a huge difference, because Poetry need to download and install all packages just to get their version and only after that can it resolve dependencies.
- jstrong 4y ago
  I don't understand - if it didn't know what version until after download/install - which version would it download/install?
  - kissgyorgy 4y ago
    Oh sorry, I was talking about dependency resolution, OP was talking about install :/
  - ywain 4y ago
    If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. So when you have many packages with many dependencies, which themselves have dependencies etc. it can take a while for poetry to assemble the full dependency graph and determine whether there are any unsolvable constraints (e.g. package foo depends on package bar with version >= 2, but package baz depends on package bar with version < 2). Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package?
    - hobofan 4y ago
      > If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. It's even worse than that. It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. As that script may contain arbitrary platform-dependent logic (and in the case of ML-related packages often does), it can be impossible to resolve dependencies for other platforms. > Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package? Yes exactly. For dependency resolution, cargo uses only a git based index[0] which is optimized to contain only the information required for dependency resolution (omitting other package metadata such as e.g. authors). So it syncs the git repository and after that it is just lookups in local files of the index. Only after dependency resolution does it need to consult an external server for retrieval of the actual package contents. [0]: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index
      - BerislavLopac 4y ago
        > It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. Only for packages that use setup.py (which is still heavily used; not sure whether it's still a majority). It is slowly being replaced by setup.cfg [0] and, more recently, pyproject.toml [1], which both contain dependencies in a declarative format. [0] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c... [1] https://peps.python.org/pep-0631/ https://peps.python.org/pep-0631/
    - deleted 4y ago
      [deleted]
- gazpacho 4y ago
  Yes and no: it needs to download it to discover its dependencies. The version is encoded in the wheel file name.
williamvds 4y ago
Last time I used Poetry, I got frustrated at how long updating the lock file (i.e. resolving dependencies) took. Adding the debug flag showed it was spending most of its time combing through each version of the setuptools package, oldest to newest, to pick the most recent compatible version. I was surprised it didn't do any caching, or possibly try binary search, though the latter could be rather imprecise. Sure, cargo being written in Rust might give the dependency resolver a nice baseline speed boost, but optimisations matter too.
- frafra 4y ago
  pdm is almost a drop-in replacement for Poetry, while being much faster on solving dependencies; look for "A Review: Pipenv vs. Poetry vs. PDM" if interested, or try it on your project.
  - 5e92cb50239222b 4y ago
    https://dev.to/frostming/a-review-pipenv-vs-poetry-vs-pdm-39b4 https://dev.to/frostming/a-review-pipenv-vs-poetry-vs-pdm-39... https://news.ycombinator.com/item?id=29502715 https://news.ycombinator.com/item?id=29502715
  - SAI_Peregrinus 4y ago
    That, really, demonstrates the cultural problem Python packaging has. With Rust, if you have a packaging improvement you make a Cargo plugin or you write an RFC & get your changes added to Cargo itself. So the Rust package manager is Cargo, and you don't have an endless string of alternatives to pick among. Just keep using Cargo, and it keeps getting better.

Cargo does semantic versioning-based dependency resolution, similar to Poetry, it's just a lot faster because it's written in a Rust. I don't think there's a big difference in the work that happens.

kissgyorgy 4y ago

I don't know anything about Rust, but probably there is a huge difference, because Poetry need to download and install all packages just to get their version and only after that can it resolve dependencies.

jstrong 4y ago
I don't understand - if it didn't know what version until after download/install - which version would it download/install?
- kissgyorgy 4y ago
  Oh sorry, I was talking about dependency resolution, OP was talking about install :/
- ywain 4y ago
  If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. So when you have many packages with many dependencies, which themselves have dependencies etc. it can take a while for poetry to assemble the full dependency graph and determine whether there are any unsolvable constraints (e.g. package foo depends on package bar with version >= 2, but package baz depends on package bar with version < 2). Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package?
  - hobofan 4y ago
    > If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. It's even worse than that. It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. As that script may contain arbitrary platform-dependent logic (and in the case of ML-related packages often does), it can be impossible to resolve dependencies for other platforms. > Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package? Yes exactly. For dependency resolution, cargo uses only a git based index[0] which is optimized to contain only the information required for dependency resolution (omitting other package metadata such as e.g. authors). So it syncs the git repository and after that it is just lookups in local files of the index. Only after dependency resolution does it need to consult an external server for retrieval of the actual package contents. [0]: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index
    - BerislavLopac 4y ago
      > It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. Only for packages that use setup.py (which is still heavily used; not sure whether it's still a majority). It is slowly being replaced by setup.cfg [0] and, more recently, pyproject.toml [1], which both contain dependencies in a declarative format. [0] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c... [1] https://peps.python.org/pep-0631/ https://peps.python.org/pep-0631/
  - deleted 4y ago
    [deleted]
gazpacho 4y ago
Yes and no: it needs to download it to discover its dependencies. The version is encoded in the wheel file name.

williamvds 4y ago

Last time I used Poetry, I got frustrated at how long updating the lock file (i.e. resolving dependencies) took. Adding the debug flag showed it was spending most of its time combing through each version of the setuptools package, oldest to newest, to pick the most recent compatible version. I was surprised it didn't do any caching, or possibly try binary search, though the latter could be rather imprecise. Sure, cargo being written in Rust might give the dependency resolver a nice baseline speed boost, but optimisations matter too.

frafra 4y ago
pdm is almost a drop-in replacement for Poetry, while being much faster on solving dependencies; look for "A Review: Pipenv vs. Poetry vs. PDM" if interested, or try it on your project.
- 5e92cb50239222b 4y ago
  https://dev.to/frostming/a-review-pipenv-vs-poetry-vs-pdm-39b4 https://dev.to/frostming/a-review-pipenv-vs-poetry-vs-pdm-39... https://news.ycombinator.com/item?id=29502715 https://news.ycombinator.com/item?id=29502715
- SAI_Peregrinus 4y ago
  That, really, demonstrates the cultural problem Python packaging has. With Rust, if you have a packaging improvement you make a Cargo plugin or you write an RFC & get your changes added to Cargo itself. So the Rust package manager is Cargo, and you don't have an endless string of alternatives to pick among. Just keep using Cargo, and it keeps getting better.