4 ms·

If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. So when you have many packages with many dependencies, whi

by ywain 4y ago

If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. So when you have many packages with many dependencies, which themselves have dependencies etc. it can take a while for poetry to assemble the full dependency graph and determine whether there are any unsolvable constraints (e.g. package foo depends on package bar with version >= 2, but package baz depends on package bar with version < 2). Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package?

hobofan 4y ago
> If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. It's even worse than that. It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. As that script may contain arbitrary platform-dependent logic (and in the case of ML-related packages often does), it can be impossible to resolve dependencies for other platforms. > Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package? Yes exactly. For dependency resolution, cargo uses only a git based index[0] which is optimized to contain only the information required for dependency resolution (omitting other package metadata such as e.g. authors). So it syncs the git repository and after that it is just lookups in local files of the index. Only after dependency resolution does it need to consult an external server for retrieval of the actual package contents. [0]: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index
- BerislavLopac 4y ago
  > It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. Only for packages that use setup.py (which is still heavily used; not sure whether it's still a majority). It is slowly being replaced by setup.cfg [0] and, more recently, pyproject.toml [1], which both contain dependencies in a declarative format. [0] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c... [1] https://peps.python.org/pep-0631/ https://peps.python.org/pep-0631/
deleted 4y ago
[deleted]

by ywain 4y ago

hobofan 4y ago

> If I'm not mistaken, it needs to download a package to know its dependencies and version constraints. It's even worse than that. It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. As that script may contain arbitrary platform-dependent logic (and in the case of ML-related packages often does), it can be impossible to resolve dependencies for other platforms. > Not sure how other package managers avoid that. Maybe the central package repositories can expose the dependencies metadata without needing to download the actual package? Yes exactly. For dependency resolution, cargo uses only a git based index[0] which is optimized to contain only the information required for dependency resolution (omitting other package metadata such as e.g. authors). So it syncs the git repository and after that it is just lookups in local files of the index. Only after dependency resolution does it need to consult an external server for retrieval of the actual package contents. [0]: https://github.com/rust-lang/crates.io-index https://github.com/rust-lang/crates.io-index

BerislavLopac 4y ago
> It needs to execute a python script (setup.py?) per package to get a list of it's dependencies and constraints. Only for packages that use setup.py (which is still heavily used; not sure whether it's still a majority). It is slowly being replaced by setup.cfg [0] and, more recently, pyproject.toml [1], which both contain dependencies in a declarative format. [0] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c... [1] https://peps.python.org/pep-0631/ https://peps.python.org/pep-0631/

deleted 4y ago

[deleted]