4 ms·

Did I get it right, the problem is basically that the dependency info is turing complete and decided "at installtime", thus the dependency graph cannot be compu

Did I get it right, the problem is basically that the dependency info is turing complete and decided "at installtime", thus the dependency graph cannot be computed without installing many versions of all dependencies? (I do not use python, so I don't know much about how pip works) Are there other languages that have dependencies decided at installtime?

vmchale 6y ago
> dependency info is turing complete It's NP-complete in general I think, definitely the case in Haskell-land. I think OCaml sets an explicit timeout for dependency resolution? > the dependency graph cannot be computed without installing many versions of all dependencies? Does PyPi not have an index or something? (with statically known package bounds?)
cipherboy 6y ago
Most distros have install time dependency resolution, such as yum/dnf using libsolv. Repos are a collection of packages alongside metadata tables containing package versions and their dependencies. The tables get cached locally and are used to resolve dependencies prior to downloading any actual RPMs.
JackC 6y ago
Right, for packages distributed as source (vs. having prebuild "binary wheels") the dependencies are specified in python code. Example from one comment on the bug: setup( install_requires=[random.choice(["urllib3", "requests"])] ) This example wouldn't make any sense, but you could imagine installing different dependencies for x86 CPUs or something via a runtime check, and there are lots of packages that use this for checking python versions even though there's now a static way to do that. So that leads to this situation, from another comment: "And as an example, [the botocore package, which has releases nearly daily] depends on python-dateutil>=2.1,<3.0.0. So if [your dependency constraints are] to install python-dateutil 3.0.0 and botocore, pip will have to backtrack through every release of botocore before it can be sure that there isn't one that works with dateutil 3.0.0. ... And worse still, if an ancient version of botocore does have an unconstrained dependency on python-dateutil, we could end up installing it with dateutil 3.0.0, and have a system that, while technically consistent, doesn't actually work." Sounds like there's a long term plan that could fix this situation. Binary wheels already have the needed metadata in a way that could be exposed by PyPI via a fast "fetch dependency constraints for all versions" API, but isn't yet. And for source dists there's a very new plan ( https://www.python.org/dev/peps/pep-0643/ https://www.python.org/dev/peps/pep-0643/ ) to let them indicate that they don't modify install_requires at runtime, so their deps could also be exposed via API, but the ecosystem will have to catch up with that. I dunno what pip does in the meantime, though! (I'm a python dev but haven't followed this beyond skimming the bug, so I hope I'm getting this right.)

vmchale 6y ago

> dependency info is turing complete It's NP-complete in general I think, definitely the case in Haskell-land. I think OCaml sets an explicit timeout for dependency resolution? > the dependency graph cannot be computed without installing many versions of all dependencies? Does PyPi not have an index or something? (with statically known package bounds?)

cipherboy 6y ago

Most distros have install time dependency resolution, such as yum/dnf using libsolv. Repos are a collection of packages alongside metadata tables containing package versions and their dependencies. The tables get cached locally and are used to resolve dependencies prior to downloading any actual RPMs.

JackC 6y ago

Right, for packages distributed as source (vs. having prebuild "binary wheels") the dependencies are specified in python code. Example from one comment on the bug: setup( install_requires=[random.choice(["urllib3", "requests"])] ) This example wouldn't make any sense, but you could imagine installing different dependencies for x86 CPUs or something via a runtime check, and there are lots of packages that use this for checking python versions even though there's now a static way to do that. So that leads to this situation, from another comment: "And as an example, [the botocore package, which has releases nearly daily] depends on python-dateutil>=2.1,<3.0.0. So if [your dependency constraints are] to install python-dateutil 3.0.0 and botocore, pip will have to backtrack through every release of botocore before it can be sure that there isn't one that works with dateutil 3.0.0. ... And worse still, if an ancient version of botocore does have an unconstrained dependency on python-dateutil, we could end up installing it with dateutil 3.0.0, and have a system that, while technically consistent, doesn't actually work." Sounds like there's a long term plan that could fix this situation. Binary wheels already have the needed metadata in a way that could be exposed by PyPI via a fast "fetch dependency constraints for all versions" API, but isn't yet. And for source dists there's a very new plan ( https://www.python.org/dev/peps/pep-0643/ https://www.python.org/dev/peps/pep-0643/ ) to let them indicate that they don't modify install_requires at runtime, so their deps could also be exposed via API, but the ecosystem will have to catch up with that. I dunno what pip does in the meantime, though! (I'm a python dev but haven't followed this beyond skimming the bug, so I hope I'm getting this right.)