5 ms·

The real thing that I hope someone is able to solve is downloading such huge amounts of unnecessary code. As I understand, the bulk of the torch binary is just

The real thing that I hope someone is able to solve is downloading such huge amounts of unnecessary code. As I understand, the bulk of the torch binary is just a huge nvfatbin compiled for every SM under the sun when you usually just want it to run on whatever accelerators you have on hand. Even just making narrow builds of like `pytorch-sm120a` (with stuff like cuBLAS thin binaries paired with it too) as part of a handy uv extra or something like that would make it much quicker and easier.

mikepurvis 1y ago
Another piece is that PyPI has no index— it's just a giant list of URLs [1] where any required metadata (eg, the OS, python version, etc) is encoded in the filename. That makes it trivial to throw behind a CDN since it's all super static, but it has some important limitations: - there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) - there's no way to get hashes of the archives - there's no way to do things like reverse-search (show me everything that depends on x) I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc. Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it. [1]: Top-level: https://pypi.org/simple/ https://pypi.org/simple/ Individual package: https://pypi.org/simple/pyyaml/ https://pypi.org/simple/pyyaml/
- zahlman 1y ago
  Your information is out of date. > there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) Not true for wheels; PyPI implements https://peps.python.org/pep-0658/ https://peps.python.org/pep-0658/ here. You can pre-download just the dependency info instead. For sdists, this is impossible until we can drop support for a bunch of older packages that don't follow modern standards (which is to say, including the actual "built" metadata as a PKG-INFO file, and having that file include static data for at least name, version and dependencies). I'm told there are real-world projects out there for which this is currently impossible, because the dependencies... depend on things that can't be known without inspecting the end user's environment. At any rate, this isn't a PyPI problem. > there's no way to get hashes of the archives This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/ https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links. But yes, there is no reverse-search support.
  - mikepurvis 1y ago
    Ah interesting, thanks for that! I was frustrated once again recently to note that `pip install --dry-run` required me to pre-download all packages, so I assumed nothing had changed.
    - zahlman 1y ago
      You could do worse than to start using --only-binary=:all: by default. (It's even been proposed as default behaviour: https://github.com/pypa/pip/issues/9140 https://github.com/pypa/pip/issues/9140) Even if you can't actually install that way, it will point out the places where sdists are needed. In principle, separate metadata availability should still at least be possible for most sdists eventually. But I'm not the one calling the shots here.
      - mikepurvis 1y ago
        For clarity, if I do pip install -r requirements.txt --dry-run --only-binary=:all:` Should I expect that to download only metadata and not whole wheels/sdists for everything? Or does that depend on everything in my requirements file being available as a wheel?
        zahlman 1y ago
        If there are things not available as a wheel, it should just fail (and highlight what failed).
  - aidenn0 1y ago
    >> there's no way to get hashes of the archives > This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/ https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links. Also, this is an implementation detail, but the archives are stored in a content-addressable store with the path being based on the (hexadecimal) blake2b/256 hash of their contents.

mikepurvis 1y ago

Another piece is that PyPI has no index— it's just a giant list of URLs [1] where any required metadata (eg, the OS, python version, etc) is encoded in the filename. That makes it trivial to throw behind a CDN since it's all super static, but it has some important limitations: - there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) - there's no way to get hashes of the archives - there's no way to do things like reverse-search (show me everything that depends on x) I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc. Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it. [1]: Top-level: https://pypi.org/simple/ https://pypi.org/simple/ Individual package: https://pypi.org/simple/pyyaml/ https://pypi.org/simple/pyyaml/

zahlman 1y ago
Your information is out of date. > there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) Not true for wheels; PyPI implements https://peps.python.org/pep-0658/ https://peps.python.org/pep-0658/ here. You can pre-download just the dependency info instead. For sdists, this is impossible until we can drop support for a bunch of older packages that don't follow modern standards (which is to say, including the actual "built" metadata as a PKG-INFO file, and having that file include static data for at least name, version and dependencies). I'm told there are real-world projects out there for which this is currently impossible, because the dependencies... depend on things that can't be known without inspecting the end user's environment. At any rate, this isn't a PyPI problem. > there's no way to get hashes of the archives This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/ https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links. But yes, there is no reverse-search support.
- mikepurvis 1y ago
  Ah interesting, thanks for that! I was frustrated once again recently to note that `pip install --dry-run` required me to pre-download all packages, so I assumed nothing had changed.
  - zahlman 1y ago
    You could do worse than to start using --only-binary=:all: by default. (It's even been proposed as default behaviour: https://github.com/pypa/pip/issues/9140 https://github.com/pypa/pip/issues/9140) Even if you can't actually install that way, it will point out the places where sdists are needed. In principle, separate metadata availability should still at least be possible for most sdists eventually. But I'm not the one calling the shots here.
    - mikepurvis 1y ago
      For clarity, if I do pip install -r requirements.txt --dry-run --only-binary=:all:` Should I expect that to download only metadata and not whole wheels/sdists for everything? Or does that depend on everything in my requirements file being available as a wheel?
      - zahlman 1y ago
        If there are things not available as a wheel, it should just fail (and highlight what failed).
- aidenn0 1y ago
  >> there's no way to get hashes of the archives > This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/ https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links. Also, this is an implementation detail, but the archives are stored in a content-addressable store with the path being based on the (hexadecimal) blake2b/256 hash of their contents.