5 ms·

The analysis contains an error. Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. That's also quite clearly seen

The analysis contains an error. Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. That's also quite clearly seen when after a phase of exponential growth, the binary artifacts still only account for 75%. So this whole strategy (actually a pitch for zigbuild) would ideally reduce the storage requirements to 25% - which would buy the whole system maybe a year or two if the growth continues. Of course, it's a good idea to build client-side. Especially considering the security implications. But it won't fundamentally change the problem.

chippiewill 2y ago
> Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. I think what the author was getting at was the combinatoric explosion as you add different variants (CPU architecture, operating systems, python versions) - but I agree that "exponential growth" is not the right term to use here.
kristoff_it 2y ago
The problem is not space, it's the fact that bandwidth costs 4x the total operating income that the PSF has. Everything else is context to understand and offer some insight into this problem. This is a pitch for Zig as a C/C++ build system because, as you can see in other comments in this submission, there's still a lot of people that don't even believe this is a solvable problem at all, while in reality Zig does a pretty good job at solving it. Only somebody actively involved with Python can really say if and how this could help improve the situation for PyPI, but if the people who actually happen to be working on something potentially beneficial weren't allowed to speak "because it's a pitch" then what even is the point of software interoperability. As an occasional Python user, I also think that it's silly that people keep putting containers everywhere and pre-building a ton of binaries when I know for a fact that you could trivially `zig biuld` that shit, since I do it in my projects all the time (see https://github.com/allyourcodebase/ https://github.com/allyourcodebase/).
- zahlman 2y ago
  >it's the fact that bandwidth costs 4x the total operating income that the PSF has. If you're getting this from my analysis included in https://news.ycombinator.com/item?id=41586751 https://news.ycombinator.com/item?id=41586751, I would encourage you to consider some of the replies to my comment. I cited and worked with AWS retail rates because that's the information I had available. It's still a major contribution on Fastly's part which likely dwarfs whatever Microsoft and Google are offering. For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example. There's really no good reason why annual downloads of Setuptools (or really anything else) should rival the population of the Earth (https://pypistats.org/packages/setuptools https://pypistats.org/packages/setuptools) - Pip caches downloaded wheels, and separate copies of Pip should ordinarily share a cache on a given machine. But I wouldn't be at all surprised to learn the Docker, k8s etc. defeat that caching scheme.
  - choeger 2y ago
    > For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example I agree to that hypothesis. But other distributions (say, Debian) should be facing the very same problem, right? How often is Python downloaded from Deb mirrors nowadays?

chippiewill 2y ago

> Binary artifacts don't cause exponential growth in storage requirements. It's still just linear. I think what the author was getting at was the combinatoric explosion as you add different variants (CPU architecture, operating systems, python versions) - but I agree that "exponential growth" is not the right term to use here.

kristoff_it 2y ago

The problem is not space, it's the fact that bandwidth costs 4x the total operating income that the PSF has. Everything else is context to understand and offer some insight into this problem. This is a pitch for Zig as a C/C++ build system because, as you can see in other comments in this submission, there's still a lot of people that don't even believe this is a solvable problem at all, while in reality Zig does a pretty good job at solving it. Only somebody actively involved with Python can really say if and how this could help improve the situation for PyPI, but if the people who actually happen to be working on something potentially beneficial weren't allowed to speak "because it's a pitch" then what even is the point of software interoperability. As an occasional Python user, I also think that it's silly that people keep putting containers everywhere and pre-building a ton of binaries when I know for a fact that you could trivially `zig biuld` that shit, since I do it in my projects all the time (see https://github.com/allyourcodebase/ https://github.com/allyourcodebase/).

zahlman 2y ago
>it's the fact that bandwidth costs 4x the total operating income that the PSF has. If you're getting this from my analysis included in https://news.ycombinator.com/item?id=41586751 https://news.ycombinator.com/item?id=41586751, I would encourage you to consider some of the replies to my comment. I cited and worked with AWS retail rates because that's the information I had available. It's still a major contribution on Fastly's part which likely dwarfs whatever Microsoft and Google are offering. For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example. There's really no good reason why annual downloads of Setuptools (or really anything else) should rival the population of the Earth (https://pypistats.org/packages/setuptools https://pypistats.org/packages/setuptools) - Pip caches downloaded wheels, and separate copies of Pip should ordinarily share a cache on a given machine. But I wouldn't be at all surprised to learn the Docker, k8s etc. defeat that caching scheme.
- choeger 2y ago
  > For what it's worth, I suspect those containerized builds are responsible for the overwhelming majority of Setuptools downloads, for example I agree to that hypothesis. But other distributions (say, Debian) should be facing the very same problem, right? How often is Python downloaded from Deb mirrors nowadays?