6 ms·

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

• orf 4y ago
  Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling
  • korijn 4y ago
    Yeah. Well, mandating wheels and getting rid of setup.py at least avoids having to run scripts, and indeed enables the next step which would be indexing all the metadata and exposing it through an API. I just thought it wouldn't necessarily be obvious to all readers of your comment.
    • orf 4y ago
      Just to be clear, package metadata already is sort of available through the pypi json api. I've got the entire set of all package metadata here: https://github.com/orf/pypi-data https://github.com/orf/pypi-data $ gzcat release_data/c/d/cdklabs.cdk-hyperledger-fabric-network.json.gz | jq '. | to_entries | .[].value.info.requires_dist' | head [ "typeguard (~=2.13.3)", "publication (>=0.0.3)", "jsii (<2.0.0,>=1.63.2)", "constructs (<11.0.0,>=10.0.5)", "aws-cdk-lib (<3.0.0,>=2.33.0)" ] It's just not everything has it, and there isn't a way to differenciate between "missing" and "no dependencies". And it's also only for the `dist` releases. But anyway, poetry uses this information during dependency resolution.
      • korijn 4y ago
        I'm aware! The issue is the mixed content (wheels, and... not wheels) on pypi. If the data is incomplete, it's useless in the sense that you're never going to be able to guarantee good results.
  • LukeShu 4y ago
    If the package is available as a wheel, you don't need to execute code to see what the requirements are; you just need to parse the "METADATA" file. However, the only way to get the METADATA for a wheel (using PyPA standard APIs, anyway) is to download the whole wheel. For comparison, pacman (the Arch Linux package manager) packages have fairly similar ".PKGINFO" file in them; but in order to support resolving dependencies without downloading the packages, the server's repository index includes not just a listing of the (name, version) tuple for each package, it also includes each package's full .PKGINFO. Enhancing the PyPA "Simple repository API" to allow fetching the METADATA independently of the wheel would be a relatively simple enhancement that would make a big difference. ---- As I was writing this comment, I discovered that PyPA did this; they adopted PEP 658 in March of this year! https://github.com/pypa/packaging.python.org/commit/1ebb57b7a7bdb38e5fec5e80262aa4bb8473d020 https://github.com/pypa/packaging.python.org/commit/1ebb57b7...
    • jbylund 4y ago
      Pip can use range requests to fetch just a part of the wheel, and lift the metadata out of that. So it can sometimes avoid downloading the entire wheel just to get the deps. Some package servers don't support this though. Also, there's a difference between a pep being adopted and that pep being implemented (usually a bunch of elbow grease). That said there are a couple exciting steps towards 658 being implemented: https://github.com/pypa/pip/pull/11111 https://github.com/pypa/pip/pull/11111 (just approved yesterday, not yet merged) https://github.com/pypi/warehouse/issues/8254 https://github.com/pypi/warehouse/issues/8254 (been open for forever, but there has been incremental progress made. Warehouse seems to not attract the same amount of contribution as pip)
• dalke 4y ago
  What if I have a dependency on a commercial third-party Python package which is on Conda but not on PyPI?
  • mistrial9 4y ago
    you are placing open code in a vendor lock-in, to start
    • dalke 4y ago
      Yes, I understand that. I see I misunderstood korijn's comment. My earlier reply is off-topic, so I won't continue further off the track.