9 ms·

The Python Package Index Should Get Rid of Its Training Wheels

92 points by AndyKelley 2y ago

faustin 2y ago
conda-forge handles the first part of this (reproducible builds) for most common platforms. The idea of rebuilding deleted artifacts on demand sounds nice in theory, but it has the complication that rebuilding something that depends on several other somethings will likely trigger a build cascade where a bunch of stuff has to get built in order. Hopefully none of those ancient build scripts require external resources hosted at dead links!
- AndyKelley 2y ago
  One strategy would be making PyPI packages fetch any external resources from PyPI, or at least add PyPI URLs as mirrors for such resources.
- aragilar 2y ago
  Also, this is very much assuming that the code is both C or C++ and that LLVM is the right compiler to use. Fortran is still a major part of the ecosystem, which the Zig compiler isn't going to solve. There already exists numerous options to provide compilers to the problematic platform, the fact is binary wheels (mostly) solve the issue far better than doing local builds. Also, the large packages are typically due to the need to support the huge number of possible GPU combinations (because you care about what CUDA versions are supported). This feels like a solution being forced on a problem (not that zig cc isn't cool), but the post has really misunderstood the issues around wheels.
yaleman 2y ago
The fact that tensorflow takes up 12.9TiB is truly horrifying, and most of that because they use pypi's storage as a dumping ground for their pre-release packages. What a nightmare they've put on other people's shoulders.
- aragilar 2y ago
  I believe tensorflow does remove old pre-releases (I know other projects do), so that number I think might be fairly static? That tensorflow is that big isn't surprising, given the install of it plus its dependencies is many gigabytes (you can see the compressed sizes of wheels on the release pages e.g. https://pypi.org/project/tensorflow/#files https://pypi.org/project/tensorflow/#files), and the "tensorflow" package (as opposed to the affiliated packages) based on https://py-code.org/stats https://py-code.org/stats is 965.7 GiB, which really only includes a relatively small number of pre-releases. Why tenserflow is that big comes down to needing to support many different kinds of GPUs with different ecosystem versions, and I suspect the build time of them with zig cc (assuming it works, and doesn't instead require pulling in a different compiler/toolchain) would be so excessive (especially on IoT/weaker devices) that it would make the point of the exercise moot.
- amoshebb 2y ago
  Is it though? If it saves one engineer one afternoon that storage has paid for itself, and this thing has hundreds of thousands of downloads a day. Wouldn’t it be more horrifying to force everybody who wants to use a prerelease to waste an afternoon getting it to build just to save half a hard drive?
  - skeledrew 2y ago
    That's besides the point though. Yes, having prebuilt binaries is very helpful. But what happens if Fastly decides against renewing next time and there is nobody else willing to sponsor? The cost is through the sky for the PSF to handle. Where does PyPI go?
    - zahlman 2y ago
      The issue there is how much gets downloaded from PyPI, not how much storage space it takes up. Making an archive copy of all of PyPI would only require a handful of ordinary drives. But the most popular packages get on the order of ten million daily downloads each. When someone downloads and installs a package, it's generally a single wheel or sdist, out of a potentially massive version/platform matrix for that project. That inflates the storage cost, but it isn't why the bandwidth requirements are so high. A ton of CI systems are apparently poorly designed, individual wheels are bloated, and we can't use the best available compression. Those are the biggest issues.
rightbyte 2y ago
These package respositories are used in a wasteful way. Probably by thousands of CI servers spinning up blank slate docker containers etc.
- strokirk 2y ago
  CI providers should definitely start proxying PyPI with their own cache.
  - zahlman 2y ago
    Pip already does its own caching, but it's maddeningly difficult to even locate and extract files, let alone set up anything usable. It's also needlessly difficult to make Pip just use such a cache directly - for example, if you haven't pinned a version, it will automatically check PyPI to figure out the latest version even if you have cached wheels already. I don't know (I lack the experience), but I assume that container systems get in the way of Pip finding its normal cache, too. (If they're emulating a filesystem or something, then the cache is in the wrong filesystem unless you reuse the container.)
  - Spivak 2y ago
    It might honestly be easier and cheaper for those CI providers to just pay PyPI's bandwidth costs.
  - Numerlor 2y ago
    I wanted to spin up a mirror locally to do simple caching for docker builds but the tooling was lacking, there was a way to do a direct mirror of pypi locally but no other way of adding custom indices
    - 0x138d5 2y ago
      Artifactory?
    - TheChaplain 2y ago
      DevPi might be your answer I think. Couple of years ago I set it up as a proxy plus hosting my own packages locally.
choeger 2y ago
The analysis contains an error. Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. That's also quite clearly seen when after a phase of exponential growth, the binary artifacts still only account for 75%. So this whole strategy (actually a pitch for zigbuild) would ideally reduce the storage requirements to 25% - which would buy the whole system maybe a year or two if the growth continues. Of course, it's a good idea to build client-side. Especially considering the security implications. But it won't fundamentally change the problem.
- chippiewill 2y ago
  > Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. I think what the author was getting at was the combinatoric explosion as you add different variants (CPU architecture, operating systems, python versions) - but I agree that "exponential growth" is not the right term to use here.
- kristoff_it 2y ago
  The problem is not space, it's the fact that bandwidth costs 4x the total operating income that the PSF has. Everything else is context to understand and offer some insight into this problem. This is a pitch for Zig as a C/C++ build system because, as you can see in other comments in this submission, there's still a lot of people that don't even believe this is a solvable problem at all, while in reality Zig does a pretty good job at solving it. Only somebody actively involved with Python can really say if and how this could help improve the situation for PyPI, but if the people who actually happen to be working on something potentially beneficial weren't allowed to speak "because it's a pitch" then what even is the point of software interoperability. As an occasional Python user, I also think that it's silly that people keep putting containers everywhere and pre-building a ton of binaries when I know for a fact that you could trivially `zig biuld` that shit, since I do it in my projects all the time (see https://github.com/allyourcodebase/ https://github.com/allyourcodebase/).
  - zahlman 2y ago
    >it's the fact that bandwidth costs 4x the total operating income that the PSF has. If you're getting this from my analysis included in https://news.ycombinator.com/item?id=41586751 https://news.ycombinator.com/item?id=41586751, I would encourage you to consider some of the replies to my comment. I cited and worked with AWS retail rates because that's the information I had available. It's still a major contribution on Fastly's part which likely dwarfs whatever Microsoft and Google are offering. For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example. There's really no good reason why annual downloads of Setuptools (or really anything else) should rival the population of the Earth (https://pypistats.org/packages/setuptools https://pypistats.org/packages/setuptools) - Pip caches downloaded wheels, and separate copies of Pip should ordinarily share a cache on a given machine. But I wouldn't be at all surprised to learn the Docker, k8s etc. defeat that caching scheme.
Havoc 2y ago
12TB for tensorflow is absurd.
zamlag 2y ago
At work we don't use PyPI any longer. We have our own set of curated packages, the security issues are just too great: https://developers.slashdot.org/story/24/09/15/0030229/fake-python-coding-tests-installed-malicious-software-packages-from-north-korea https://developers.slashdot.org/story/24/09/15/0030229/fake-... https://jfrog.com/blog/revival-hijack-pypi-hijack-technique-exploited-22k-packages-at-risk/ https://jfrog.com/blog/revival-hijack-pypi-hijack-technique-... https://jfrog.com/blog/leaked-pypi-secret-token-revealed-in-binary-preventing-suppy-chain-attack/ https://jfrog.com/blog/leaked-pypi-secret-token-revealed-in-... We consider switching to Java, C++ or Rust because of general quality issues with Python.
- notpushkin 2y ago
  What do Java and Rust (and your C++ package manager of choice) do to mitigate those things?
  - ajrqh 2y ago
    What is not clear about "general quality issues", i.e., issues unrelated to package management?
    - notpushkin 2y ago
      OK, my bad. It is even more unclear though – do you have any particular examples of such issues?
  - deleted 2y ago
    [deleted]
  - theamk 2y ago
    Don't know about Java or Rust, but in C++ it's much harder to get new packages, which works wonder to keep number of dependencies down. In Python, installing packages is so simple, people just do it after 10 second google search - and that library can pull dozens or hundreds of dependencies with usually no review. In C++, given that you have to manually find the library and incorporate into the build system, people generally spend a few minutes looking at the options and choosing the best one - and this includes checking for things "how long has this library been around", "does it have many users" and "is it in healthy state". And this must be repeated for each dependency as well, so that even a dozen of dependencies is a huge negative - and such library will not be used unless there are no better alternatives. (The exception to this rule are libraries provided by your Linux distribution. Those can be easily installed by the dozen, and that's OK - the distribution makers did all the hard work for you, vetting and packaging those libraries) This in general means a much healthier dependency state for C++, as well as much higher code quality. No one is going to add a dependency to a core library just to add a better progress bars for example.
zzzeek 2y ago
My own personal TL;DR would be, pypi has to store too much data in the form of pre built binaries that are uploaded by package authors. Python should adopt a repeatable build format so that PyPi itself can build wheels for any platform on demand (edit: am I misunderstanding? did they mean, wheels can be built as part of the local install process?). Author is involved with some special compiler to do this. Personally I'd love if PyPi could build our wheels for us. That would be great, we use GitHub actions right now which has its own complexities and for years we had nothing. But that would mean a huge ramp up in processing capability for PyPi. Considering PyPi can't even handle having its packages signed and "solved" the problem by sending authors obnoxious emails if we even dare to push up a signature file, im not too optimistic for such a change vs. their just continuing to rely on corporate sponsors to deliver bandwidth.
- robertlagrant 2y ago
  My read was it enables both: PyPI can delete all packages that haven't been downloaded in a year or more, safe in the knowledge that they can be quickly recreated and cached, but also that more packages can be downloaded as source, and compiled at target.
eesmith 2y ago
Some issues I see are: - packages which only distribute binaries (eg, closed-source or source-for-a-fee distributions) - it looks like Zig's C compiler does not support OpenMP, which I use - what is the cut-off time for source vs. binary distribution? My package takes about a minute to compile (it has a lot of auto-generated source). - what's the user impact if they have 10 projects which are just under that threshold? - compile-time dependencies which are not recorded in pyproject.toml (like, having a Fortran compiler, having yacc/bison, etc.)
rwmj 2y ago
Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.
- atemerev 2y ago
  Nope. System packages and language dependencies are better not to be mixed, and the horrors if you do that are very real.
- zahlman 2y ago
  There are many problems with this idea. * Windows exists. * The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian). * There simply are far too many packages out there to make it workable. * Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source. * Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.
  - cozzyd 2y ago
    Yeah but do you really need a separate copy of numpy/matplotlib/scipy/etc. for every single project? I don't have infinite disk space...
    - zahlman 2y ago
BiteCode_dev 2y ago
One of the reasons Python is so popular as a scripting language in science and ML is that it has a very good story for installing Frankenstein code bases made of assembly, C and Pascal sprinkled with SIMD. I was here before Anaconda popularized the idea of binary packages for Python and inspired wheels to replace eggs, and I don't want to go back to having to compile that nightmare on my machine. People that have that kind of idea are likely capable of running kubs containers, understand vectorization and can code a monad on the top of their head. Half of the Python coders are struggling to use their terminal. You have Windows dev that lives in Visual Studio, teachers of high school that barely show a few functions, mathematicians that are replacing R/Matlab, biologists forced to script something to write a paper, frontend dev that just learned JS is not the only language, geographers that are supplicating their GIS system to do something it's not made for, kids messing with their dad laptop, and probably a dog somewhere. Binary wheels are a gift from the Gods.
- RockRobotRock 2y ago
  If I didn’t have access to binary wheels, I wouldn’t have been able to learn Python in high school as a heavy Windows user. Even now, running into package compile issues is a sure fire way to lose half an hour of my time.
- nektro 2y ago
  the article does not suggest removing wheeels
- HelloNurse 2y ago
  Compiling Python extensions is a nightmare because we allow Autoconf, CMake, Visual Studio, Bazel etc. to make it complicated and nonportable; when someone sets out to wrap some library for Python the quality of the result is limited by the quality of the tools and by low expectations. A serious engineering effort along the lines of the Zig compiler would allow Python to build almost everything from source out of the box; exotic compilers and binary dependencies, not "Frankenstein code bases" per se, are the actual obstacles.
  - 2y ago
Siecje 2y ago
Compatible releases should replace older versions. Why would you want to install an old version of ruff?
- zahlman 2y ago
  >Why would you want to install an old version of ruff? Because something else in your project is broken by the new version.
- TheCleric 2y ago
  Because packages shouldn't be upgraded in a project until they have been tested. If you give me no way of downloading version 1.0 of a package in my project because 1.1 supersedes it and 1.1 breaks my project, what am I supposed to do?
Wowfunhappy 2y ago
I'm a bit confused as to why this costs so much. I thought storage was cheap? Bandwidth is more expensive, but shouldn't be relevant to this problem. It doesn't matter whether 5 people request the same binary or 5 people request 5 different binaries for different platforms, if all the binaries are 1 gb you're transferring 5 gb of data either way.
- bravetraveler 2y ago
  Really depends on how you get the storage/interface with it. Cloud/VPS is some of the worst 'bang for the buck' in my experience, where S3 or dedicated can be more favorable at a point. Not all gigglebytes cost the same :) The article may shed some light on this, I'm not aware. Haven't read yet! I've managed a few RPM package mirrors. Dedicated gear that's a few generations old with no-name providers was the best experience.
- kristoff_it 2y ago
  Once you have control over generating the binaries, you can create a system that is more akin to what Nix offers, where other parties can setup secondary caches for their projects, and if they don't the main registry has the option to decide what to do about it without breaking any package. Currently PyPI is railroaded into a fairly tight set of choices because it can't exercise a lot of control over binary data. The blog post mentions this point directly.
  - Wowfunhappy 2y ago
    Thanks. The blog post alludes to mirrors but I didn't realize that was the primary goal of this project. I am somewhat skeptical that these projects will have an incentive to actually set up mirrors. For example, I imagine Tensorflow likes getting free hosting.
    - kristoff_it 2y ago
      I haven't checked, but usually when it's this visible, big tech companies tend to reciprocate so I would be surprised if Google didn't sponsor the PSF in any way. That said, once deleting prebuilt binaries doesn't break the package anymore, if Google were to not play fair, PyPI could simply delete them. Users would experience temporary discomfort (but nothing would be irremediably broken), and soon after I'm sure that Google would decide to setup a cache for TF. PyPI would almost certainly not even need to actually do it, the implication that it could be done would probably be enough to align everybody towards what's best for the ecosystem.
trlampert 2y ago
"When Python came into existence, repeatable builds (i.e. not yet reproducible, but at least correctly functioning on more than one machine) were a pipe dream. Building C/C++ projects reliably has been an intractable problem for a long time, but that's not true anymore." I'd dispute that. It used to be the case that building NumPy just worked, now there are Cython/meson and a whole lot of other dependency issues and the build fails. "At the Zig Software Foundation we look up to the Python Software Foundation as a great example of a fellow 501(c)(3) non-profit organization that was able to grow an incredibly vibrant community ..." Better don't meet your heroes. Python was a reasonable community made up and driven by creative individuals. In the last 8 years, it has been taken over by corporate bureaucrats who take credit for the remnants of that community and who will destroy it. The PSF has never done anything except for selling expensive conference tickets and taking care of its own.
JohnMakin 2y ago
I don’t pretend to know the answer here, but python is unavoidable in my work and the packaging is a constant source of irritation for me. Disclaimer: I am not pretending to be a python expert here but coming from a C background this anecdote is baffling to me. I am writing some lambda OAuth glue logic using python because python is the best choice for this particular implementation. I need to package “jwt” library which worked absolutely fine for a while. Then I upgrade python versions and the particular pre-built container I was using to run pip and zip the packages up ended up with totally borked versions of jwt which seemed to not have functions I was previously using fine. Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. So, the solution was to specifically specify PyJWT and which version I wanted in my pip install. Great! That’s how I think it should work and I was a little baffled that pip had made that decision for me previously. Anyway, it now has my missing functions but is still crashing in my blue deployments. Wtf? Oh, this PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” (making sure to get the compatible version matrix here, I at this point had stopped trusting pip). So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple. yea, wrangling makefiles and fussing with linkers can be annoying but I’d take that any day over this bs.
- zahlman 2y ago
  > Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. Names on PyPI have no necessary correlation with names used to `import` modules in the code. This is known and by design and has always been so; and it is thoroughly documented (e.g. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut...). There are at important architectural reasons for it: * A package installed from PyPI can legally provide any number of importable packages and/or modules - including zero. * Making it work the way you imagine would require additional verification work, with no clear idea of who's responsible for that work nor what to do when the assumption is violated. * Sometimes projects have valid reasons to rename what you `import` at top level. It could be to avoid common conflicts; it could be because of internal open-source politics (see: PIL/Pillow); it could be because a legacy name was needed for some disambiguation purpose with Conda; for whatever rare reasons it might just make more logical sense. In the extreme, some name could become a keyword in a later Python version. (There's an actual `async` package on PyPI, although it was last updated 10 years ago.) In all of these cases, there's value in keeping the name used on PyPI the same. * There are valid reasons why people might want to swap implementations of the same library. The PyPI system allows you to just change the name provided to `pip install` without modifying the code. You actually ran into that last point: both `jwt` and `PyJWT` on PyPI refer to implementations of JSON Web Tokens that expect you to `import jwt` in the code. They may or may not have different interfaces and supply different functionality, and are maintained by different people. You can't seriously expect to forbid everyone else from publishing such an implementation just because one already exists. > This PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” There are a few possibilities: * The maintainer messed up and forgot to list it as a dependency in the manifest. These things happen. Other people's projects are allowed to have their own dependencies. * The maintainer listed `cryptography` as an optional dependency or "extra" - so that you don't have to grab it if you aren't using that algorithm. You're expected to read the documentation to understand what your project will need. * You really were intended to use the `jwt` package, but a new version of that project removed functionality you were depending on. Other people's projects are allowed to change their interfaces and go through deprecation cycles. In this case, you would presumably find that your code also needed to be rewritten to invoke the `cryptography` functionality explicitly/directly. >making sure to get the compatible version Pip solves dependencies for you. If you're running into version incompatibility issues, generally it's either the fault of the package maintainers, or the environment you want to create is fundamentally not possible due to conflicting version requirements of your transitive dependencies. (Pip is designed to only install one version of a given package in a given environment, because Python is designed to only `import` a given module from one place and then reuse the cached module object.) >So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple Yes, it's obvious. Yes, normally it is dead simple. Veterans commonly run into hairy problems regardless. There are plenty of faults in the system - this just isn't generally considered one of them. The scope of the problem solved (or at least undertaken) by the Pip/PyPI system is much larger than you seem to imagine. In particular, it often has to do that same wrangling with linkers and Makefiles that you describe, and in an automated fashion at that. (Python code commonly has to interoperate with extensions written in C - and FORTRAN, and other languages - which is provided as part of the same package.)
atemerev 2y ago
Nope. Wheels are the only thing that makes Python/PyPI usable. I don’t want to wait tens of minutes to recompile pytorch, or something (and conda is way too heavyweight for my tastes)
pxc 2y ago
There are already lots of passable package managers that know how to provide working binaries for the native, non-Python dependencies of Python packages. Instead of trying to make Python packages' build processes learn how to build everything else in the world, one thing Python packages could do is just record their external dependencies in a useful way. Then package managers that are actually already designed for and committed to working with multiple programming language ecosystems could handle the rest. This is something that could be used by Nix, Guix, Spack, as well as more conventional software distributions like Conda, Pkgsrc, MacPorts, Homebrew, etc. With the former, users could even set up per-project environments that contain those external dependencies, like virtualenvs but much more general. But the simple feature of this metadata would naturally be valuable, if provided well, to maintainers of all Linux distros and many other software distributions, where autogenerated packages are already the norm for languagea like Rust and Go, while creating such tooling for Python is riddled with thorny problems— o these two proposals are not mutually exclusive, and perhaps each is individually warranted on its own. Enriching package metadata in this simple way has already been proposed here: htps://peps.python.org/pep-0725/

92 points by AndyKelley 2y ago

conda-forge handles the first part of this (reproducible builds) for most common platforms. The idea of rebuilding deleted artifacts on demand sounds nice in theory, but it has the complication that rebuilding something that depends on several other somethings will likely trigger a build cascade where a bunch of stuff has to get built in order. Hopefully none of those ancient build scripts require external resources hosted at dead links!

AndyKelley 2y ago
One strategy would be making PyPI packages fetch any external resources from PyPI, or at least add PyPI URLs as mirrors for such resources.
aragilar 2y ago
Also, this is very much assuming that the code is both C or C++ and that LLVM is the right compiler to use. Fortran is still a major part of the ecosystem, which the Zig compiler isn't going to solve. There already exists numerous options to provide compilers to the problematic platform, the fact is binary wheels (mostly) solve the issue far better than doing local builds. Also, the large packages are typically due to the need to support the huge number of possible GPU combinations (because you care about what CUDA versions are supported). This feels like a solution being forced on a problem (not that zig cc isn't cool), but the post has really misunderstood the issues around wheels.

yaleman 2y ago

The fact that tensorflow takes up 12.9TiB is truly horrifying, and most of that because they use pypi's storage as a dumping ground for their pre-release packages. What a nightmare they've put on other people's shoulders.

aragilar 2y ago
I believe tensorflow does remove old pre-releases (I know other projects do), so that number I think might be fairly static? That tensorflow is that big isn't surprising, given the install of it plus its dependencies is many gigabytes (you can see the compressed sizes of wheels on the release pages e.g. https://pypi.org/project/tensorflow/#files https://pypi.org/project/tensorflow/#files), and the "tensorflow" package (as opposed to the affiliated packages) based on https://py-code.org/stats https://py-code.org/stats is 965.7 GiB, which really only includes a relatively small number of pre-releases. Why tenserflow is that big comes down to needing to support many different kinds of GPUs with different ecosystem versions, and I suspect the build time of them with zig cc (assuming it works, and doesn't instead require pulling in a different compiler/toolchain) would be so excessive (especially on IoT/weaker devices) that it would make the point of the exercise moot.
amoshebb 2y ago
Is it though? If it saves one engineer one afternoon that storage has paid for itself, and this thing has hundreds of thousands of downloads a day. Wouldn’t it be more horrifying to force everybody who wants to use a prerelease to waste an afternoon getting it to build just to save half a hard drive?
- skeledrew 2y ago
  That's besides the point though. Yes, having prebuilt binaries is very helpful. But what happens if Fastly decides against renewing next time and there is nobody else willing to sponsor? The cost is through the sky for the PSF to handle. Where does PyPI go?
  - zahlman 2y ago
    The issue there is how much gets downloaded from PyPI, not how much storage space it takes up. Making an archive copy of all of PyPI would only require a handful of ordinary drives. But the most popular packages get on the order of ten million daily downloads each. When someone downloads and installs a package, it's generally a single wheel or sdist, out of a potentially massive version/platform matrix for that project. That inflates the storage cost, but it isn't why the bandwidth requirements are so high. A ton of CI systems are apparently poorly designed, individual wheels are bloated, and we can't use the best available compression. Those are the biggest issues.

rightbyte 2y ago

These package respositories are used in a wasteful way. Probably by thousands of CI servers spinning up blank slate docker containers etc.

strokirk 2y ago
CI providers should definitely start proxying PyPI with their own cache.
- zahlman 2y ago
  Pip already does its own caching, but it's maddeningly difficult to even locate and extract files, let alone set up anything usable. It's also needlessly difficult to make Pip just use such a cache directly - for example, if you haven't pinned a version, it will automatically check PyPI to figure out the latest version even if you have cached wheels already. I don't know (I lack the experience), but I assume that container systems get in the way of Pip finding its normal cache, too. (If they're emulating a filesystem or something, then the cache is in the wrong filesystem unless you reuse the container.)
- Spivak 2y ago
  It might honestly be easier and cheaper for those CI providers to just pay PyPI's bandwidth costs.
- Numerlor 2y ago
  I wanted to spin up a mirror locally to do simple caching for docker builds but the tooling was lacking, there was a way to do a direct mirror of pypi locally but no other way of adding custom indices
  - 0x138d5 2y ago
    Artifactory?
  - TheChaplain 2y ago
    DevPi might be your answer I think. Couple of years ago I set it up as a proxy plus hosting my own packages locally.

choeger 2y ago

The analysis contains an error. Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. That's also quite clearly seen when after a phase of exponential growth, the binary artifacts still only account for 75%. So this whole strategy (actually a pitch for zigbuild) would ideally reduce the storage requirements to 25% - which would buy the whole system maybe a year or two if the growth continues. Of course, it's a good idea to build client-side. Especially considering the security implications. But it won't fundamentally change the problem.

chippiewill 2y ago
> Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. I think what the author was getting at was the combinatoric explosion as you add different variants (CPU architecture, operating systems, python versions) - but I agree that "exponential growth" is not the right term to use here.
kristoff_it 2y ago
The problem is not space, it's the fact that bandwidth costs 4x the total operating income that the PSF has. Everything else is context to understand and offer some insight into this problem. This is a pitch for Zig as a C/C++ build system because, as you can see in other comments in this submission, there's still a lot of people that don't even believe this is a solvable problem at all, while in reality Zig does a pretty good job at solving it. Only somebody actively involved with Python can really say if and how this could help improve the situation for PyPI, but if the people who actually happen to be working on something potentially beneficial weren't allowed to speak "because it's a pitch" then what even is the point of software interoperability. As an occasional Python user, I also think that it's silly that people keep putting containers everywhere and pre-building a ton of binaries when I know for a fact that you could trivially `zig biuld` that shit, since I do it in my projects all the time (see https://github.com/allyourcodebase/ https://github.com/allyourcodebase/).
- zahlman 2y ago
  >it's the fact that bandwidth costs 4x the total operating income that the PSF has. If you're getting this from my analysis included in https://news.ycombinator.com/item?id=41586751 https://news.ycombinator.com/item?id=41586751, I would encourage you to consider some of the replies to my comment. I cited and worked with AWS retail rates because that's the information I had available. It's still a major contribution on Fastly's part which likely dwarfs whatever Microsoft and Google are offering. For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example. There's really no good reason why annual downloads of Setuptools (or really anything else) should rival the population of the Earth (https://pypistats.org/packages/setuptools https://pypistats.org/packages/setuptools) - Pip caches downloaded wheels, and separate copies of Pip should ordinarily share a cache on a given machine. But I wouldn't be at all surprised to learn the Docker, k8s etc. defeat that caching scheme.

Havoc 2y ago

12TB for tensorflow is absurd.

zamlag 2y ago

At work we don't use PyPI any longer. We have our own set of curated packages, the security issues are just too great: https://developers.slashdot.org/story/24/09/15/0030229/fake-python-coding-tests-installed-malicious-software-packages-from-north-korea https://developers.slashdot.org/story/24/09/15/0030229/fake-... https://jfrog.com/blog/revival-hijack-pypi-hijack-technique-exploited-22k-packages-at-risk/ https://jfrog.com/blog/revival-hijack-pypi-hijack-technique-... https://jfrog.com/blog/leaked-pypi-secret-token-revealed-in-binary-preventing-suppy-chain-attack/ https://jfrog.com/blog/leaked-pypi-secret-token-revealed-in-... We consider switching to Java, C++ or Rust because of general quality issues with Python.

notpushkin 2y ago
What do Java and Rust (and your C++ package manager of choice) do to mitigate those things?
- ajrqh 2y ago
  What is not clear about "general quality issues", i.e., issues unrelated to package management?
  - notpushkin 2y ago
    OK, my bad. It is even more unclear though – do you have any particular examples of such issues?
- deleted 2y ago
  [deleted]
- theamk 2y ago
  Don't know about Java or Rust, but in C++ it's much harder to get new packages, which works wonder to keep number of dependencies down. In Python, installing packages is so simple, people just do it after 10 second google search - and that library can pull dozens or hundreds of dependencies with usually no review. In C++, given that you have to manually find the library and incorporate into the build system, people generally spend a few minutes looking at the options and choosing the best one - and this includes checking for things "how long has this library been around", "does it have many users" and "is it in healthy state". And this must be repeated for each dependency as well, so that even a dozen of dependencies is a huge negative - and such library will not be used unless there are no better alternatives. (The exception to this rule are libraries provided by your Linux distribution. Those can be easily installed by the dozen, and that's OK - the distribution makers did all the hard work for you, vetting and packaging those libraries) This in general means a much healthier dependency state for C++, as well as much higher code quality. No one is going to add a dependency to a core library just to add a better progress bars for example.

zzzeek 2y ago

My own personal TL;DR would be, pypi has to store too much data in the form of pre built binaries that are uploaded by package authors. Python should adopt a repeatable build format so that PyPi itself can build wheels for any platform on demand (edit: am I misunderstanding? did they mean, wheels can be built as part of the local install process?). Author is involved with some special compiler to do this. Personally I'd love if PyPi could build our wheels for us. That would be great, we use GitHub actions right now which has its own complexities and for years we had nothing. But that would mean a huge ramp up in processing capability for PyPi. Considering PyPi can't even handle having its packages signed and "solved" the problem by sending authors obnoxious emails if we even dare to push up a signature file, im not too optimistic for such a change vs. their just continuing to rely on corporate sponsors to deliver bandwidth.

robertlagrant 2y ago
My read was it enables both: PyPI can delete all packages that haven't been downloaded in a year or more, safe in the knowledge that they can be quickly recreated and cached, but also that more packages can be downloaded as source, and compiled at target.

eesmith 2y ago

Some issues I see are: - packages which only distribute binaries (eg, closed-source or source-for-a-fee distributions) - it looks like Zig's C compiler does not support OpenMP, which I use - what is the cut-off time for source vs. binary distribution? My package takes about a minute to compile (it has a lot of auto-generated source). - what's the user impact if they have 10 projects which are just under that threshold? - compile-time dependencies which are not recorded in pyproject.toml (like, having a Fortran compiler, having yacc/bison, etc.)

rwmj 2y ago

Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.

atemerev 2y ago
Nope. System packages and language dependencies are better not to be mixed, and the horrors if you do that are very real.
zahlman 2y ago
There are many problems with this idea. * Windows exists. * The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian). * There simply are far too many packages out there to make it workable. * Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source. * Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.
- cozzyd 2y ago
  Yeah but do you really need a separate copy of numpy/matplotlib/scipy/etc. for every single project? I don't have infinite disk space...
  - zahlman 2y ago

BiteCode_dev 2y ago

One of the reasons Python is so popular as a scripting language in science and ML is that it has a very good story for installing Frankenstein code bases made of assembly, C and Pascal sprinkled with SIMD. I was here before Anaconda popularized the idea of binary packages for Python and inspired wheels to replace eggs, and I don't want to go back to having to compile that nightmare on my machine. People that have that kind of idea are likely capable of running kubs containers, understand vectorization and can code a monad on the top of their head. Half of the Python coders are struggling to use their terminal. You have Windows dev that lives in Visual Studio, teachers of high school that barely show a few functions, mathematicians that are replacing R/Matlab, biologists forced to script something to write a paper, frontend dev that just learned JS is not the only language, geographers that are supplicating their GIS system to do something it's not made for, kids messing with their dad laptop, and probably a dog somewhere. Binary wheels are a gift from the Gods.

RockRobotRock 2y ago
If I didn’t have access to binary wheels, I wouldn’t have been able to learn Python in high school as a heavy Windows user. Even now, running into package compile issues is a sure fire way to lose half an hour of my time.
nektro 2y ago
the article does not suggest removing wheeels
HelloNurse 2y ago
Compiling Python extensions is a nightmare because we allow Autoconf, CMake, Visual Studio, Bazel etc. to make it complicated and nonportable; when someone sets out to wrap some library for Python the quality of the result is limited by the quality of the tools and by low expectations. A serious engineering effort along the lines of the Zig compiler would allow Python to build almost everything from source out of the box; exotic compilers and binary dependencies, not "Frankenstein code bases" per se, are the actual obstacles.
- 2y ago

Siecje 2y ago

Compatible releases should replace older versions. Why would you want to install an old version of ruff?

zahlman 2y ago
>Why would you want to install an old version of ruff? Because something else in your project is broken by the new version.
TheCleric 2y ago
Because packages shouldn't be upgraded in a project until they have been tested. If you give me no way of downloading version 1.0 of a package in my project because 1.1 supersedes it and 1.1 breaks my project, what am I supposed to do?

Wowfunhappy 2y ago

I'm a bit confused as to why this costs so much. I thought storage was cheap? Bandwidth is more expensive, but shouldn't be relevant to this problem. It doesn't matter whether 5 people request the same binary or 5 people request 5 different binaries for different platforms, if all the binaries are 1 gb you're transferring 5 gb of data either way.

bravetraveler 2y ago
Really depends on how you get the storage/interface with it. Cloud/VPS is some of the worst 'bang for the buck' in my experience, where S3 or dedicated can be more favorable at a point. Not all gigglebytes cost the same :) The article may shed some light on this, I'm not aware. Haven't read yet! I've managed a few RPM package mirrors. Dedicated gear that's a few generations old with no-name providers was the best experience.
kristoff_it 2y ago
Once you have control over generating the binaries, you can create a system that is more akin to what Nix offers, where other parties can setup secondary caches for their projects, and if they don't the main registry has the option to decide what to do about it without breaking any package. Currently PyPI is railroaded into a fairly tight set of choices because it can't exercise a lot of control over binary data. The blog post mentions this point directly.
- Wowfunhappy 2y ago
  Thanks. The blog post alludes to mirrors but I didn't realize that was the primary goal of this project. I am somewhat skeptical that these projects will have an incentive to actually set up mirrors. For example, I imagine Tensorflow likes getting free hosting.
  - kristoff_it 2y ago
    I haven't checked, but usually when it's this visible, big tech companies tend to reciprocate so I would be surprised if Google didn't sponsor the PSF in any way. That said, once deleting prebuilt binaries doesn't break the package anymore, if Google were to not play fair, PyPI could simply delete them. Users would experience temporary discomfort (but nothing would be irremediably broken), and soon after I'm sure that Google would decide to setup a cache for TF. PyPI would almost certainly not even need to actually do it, the implication that it could be done would probably be enough to align everybody towards what's best for the ecosystem.

trlampert 2y ago

"When Python came into existence, repeatable builds (i.e. not yet reproducible, but at least correctly functioning on more than one machine) were a pipe dream. Building C/C++ projects reliably has been an intractable problem for a long time, but that's not true anymore." I'd dispute that. It used to be the case that building NumPy just worked, now there are Cython/meson and a whole lot of other dependency issues and the build fails. "At the Zig Software Foundation we look up to the Python Software Foundation as a great example of a fellow 501(c)(3) non-profit organization that was able to grow an incredibly vibrant community ..." Better don't meet your heroes. Python was a reasonable community made up and driven by creative individuals. In the last 8 years, it has been taken over by corporate bureaucrats who take credit for the remnants of that community and who will destroy it. The PSF has never done anything except for selling expensive conference tickets and taking care of its own.

JohnMakin 2y ago

I don’t pretend to know the answer here, but python is unavoidable in my work and the packaging is a constant source of irritation for me. Disclaimer: I am not pretending to be a python expert here but coming from a C background this anecdote is baffling to me. I am writing some lambda OAuth glue logic using python because python is the best choice for this particular implementation. I need to package “jwt” library which worked absolutely fine for a while. Then I upgrade python versions and the particular pre-built container I was using to run pip and zip the packages up ended up with totally borked versions of jwt which seemed to not have functions I was previously using fine. Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. So, the solution was to specifically specify PyJWT and which version I wanted in my pip install. Great! That’s how I think it should work and I was a little baffled that pip had made that decision for me previously. Anyway, it now has my missing functions but is still crashing in my blue deployments. Wtf? Oh, this PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” (making sure to get the compatible version matrix here, I at this point had stopped trusting pip). So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple. yea, wrangling makefiles and fussing with linkers can be annoying but I’d take that any day over this bs.

zahlman 2y ago
> Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. Names on PyPI have no necessary correlation with names used to `import` modules in the code. This is known and by design and has always been so; and it is thoroughly documented (e.g. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut...). There are at important architectural reasons for it: * A package installed from PyPI can legally provide any number of importable packages and/or modules - including zero. * Making it work the way you imagine would require additional verification work, with no clear idea of who's responsible for that work nor what to do when the assumption is violated. * Sometimes projects have valid reasons to rename what you `import` at top level. It could be to avoid common conflicts; it could be because of internal open-source politics (see: PIL/Pillow); it could be because a legacy name was needed for some disambiguation purpose with Conda; for whatever rare reasons it might just make more logical sense. In the extreme, some name could become a keyword in a later Python version. (There's an actual `async` package on PyPI, although it was last updated 10 years ago.) In all of these cases, there's value in keeping the name used on PyPI the same. * There are valid reasons why people might want to swap implementations of the same library. The PyPI system allows you to just change the name provided to `pip install` without modifying the code. You actually ran into that last point: both `jwt` and `PyJWT` on PyPI refer to implementations of JSON Web Tokens that expect you to `import jwt` in the code. They may or may not have different interfaces and supply different functionality, and are maintained by different people. You can't seriously expect to forbid everyone else from publishing such an implementation just because one already exists. > This PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” There are a few possibilities: * The maintainer messed up and forgot to list it as a dependency in the manifest. These things happen. Other people's projects are allowed to have their own dependencies. * The maintainer listed `cryptography` as an optional dependency or "extra" - so that you don't have to grab it if you aren't using that algorithm. You're expected to read the documentation to understand what your project will need. * You really were intended to use the `jwt` package, but a new version of that project removed functionality you were depending on. Other people's projects are allowed to change their interfaces and go through deprecation cycles. In this case, you would presumably find that your code also needed to be rewritten to invoke the `cryptography` functionality explicitly/directly. >making sure to get the compatible version Pip solves dependencies for you. If you're running into version incompatibility issues, generally it's either the fault of the package maintainers, or the environment you want to create is fundamentally not possible due to conflicting version requirements of your transitive dependencies. (Pip is designed to only install one version of a given package in a given environment, because Python is designed to only `import` a given module from one place and then reuse the cached module object.) >So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple Yes, it's obvious. Yes, normally it is dead simple. Veterans commonly run into hairy problems regardless. There are plenty of faults in the system - this just isn't generally considered one of them. The scope of the problem solved (or at least undertaken) by the Pip/PyPI system is much larger than you seem to imagine. In particular, it often has to do that same wrangling with linkers and Makefiles that you describe, and in an automated fashion at that. (Python code commonly has to interoperate with extensions written in C - and FORTRAN, and other languages - which is provided as part of the same package.)

atemerev 2y ago

Nope. Wheels are the only thing that makes Python/PyPI usable. I don’t want to wait tens of minutes to recompile pytorch, or something (and conda is way too heavyweight for my tastes)

pxc 2y ago

There are already lots of passable package managers that know how to provide working binaries for the native, non-Python dependencies of Python packages. Instead of trying to make Python packages' build processes learn how to build everything else in the world, one thing Python packages could do is just record their external dependencies in a useful way. Then package managers that are actually already designed for and committed to working with multiple programming language ecosystems could handle the rest. This is something that could be used by Nix, Guix, Spack, as well as more conventional software distributions like Conda, Pkgsrc, MacPorts, Homebrew, etc. With the former, users could even set up per-project environments that contain those external dependencies, like virtualenvs but much more general. But the simple feature of this metadata would naturally be valuable, if provided well, to maintainers of all Linux distros and many other software distributions, where autogenerated packages are already the norm for languagea like Rust and Go, while creating such tooling for Python is riddled with thorny problems— o these two proposals are not mutually exclusive, and perhaps each is individually warranted on its own. Enriching package metadata in this simple way has already been proposed here: htps://peps.python.org/pep-0725/