3 ms·

We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough Mildly surprised a service a

We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough Mildly surprised a service as big as PyPI doesn't.

bool3max 5y ago
PyPI is more complex and probably serves way more traffic.
melissalobos 5y ago
I think the idea here isn't that there isn't a PyPI mirror in Africa. It is that not everyone has internet, so this person wants to have a tiny computer output a local WiFi network people can connect to and download pip packages. Imagine a small town or village with some power and a classroom, but no internet. A teacher could setup a network and have students connect to this device and download packages so they can complete some assignment/make the next Facebook.
- bscphil 5y ago
  Or that the Internet access they do have is often metered. A friend tells me 100MB costs about 1 USD where he is, which is not an insignificant amount of money. Really puts the whole 300 MB electron app thing in perspective; at any rate it's understandable why having a PyPI mirror in the classroom would be preferable to having each student download the packages over and over.
mjw1007 5y ago
I think PyPI is mostly served via Fastly. https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...
heavyset_go 5y ago
PyPI needs more funding. PyPI even disabled their search API for the pip CLI because of infrastructure overload. It would be nice if more sponsors stepped up to fund their infrastructure.
- db65edfc7996 5y ago
  I think companies could choose to be more responsible with their usage. Looking at PyPi utilization, I have to imagine the bulk of it comes from CI/CD tooling hammering the servers without any intermediate caching.
  - heavyset_go 5y ago
    Definitely. I don't think hammering PyPI's API would be much of problem if the companies doing the hammering paid for the privilege of doing so, though.
  - sgtnoodle 5y ago
    That's definitely true. There's so many build jobs running in docker containers that lack a pip cache. I believe my company has its own package servers, but I've specifically avoided adding build system features because of that lack of caching. For one of me, there's probably a dozen people that would just hit the main pypi servers every run.
woodruffw 5y ago
> Mildly surprised a service as big as PyPI doesn't. PyPI uses Fastly as a CDN, which does indeed have presence on every continent (except the big, cold one). The problem here isn't presence, but connectivity to the outside Internet itself. Source: I'm an active developer on PyPI.
qbasic_forever 5y ago
I remember at pycon ~2016 one of the maintainers of pypi did a short talk on it and the entire pypi service at the time only ran on one or two boxes. It was surprisingly scrappy for such a critical service.
- tempay 5y ago
  This has changed a lot in recent years and is now in a much better state. Their docs give a really nice overview of the evolution over time https://warehouse.pypa.io/application.html https://warehouse.pypa.io/application.html
tentacleuno 5y ago
Serious question: should charities start downloading the entirety of NPM and PyPi (+ Node, NPM, Linux packages, etc.) onto hard drives and passing them onto schools teaching software development in third-world countries alongside the computers they normally send? I'm not sure if that's even possible, or how big the HDD would have to be to store all that, but it would be very very nice nevertheless. I can't imagine not being able to install one-off software packages.

We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough Mildly surprised a service as big as PyPI doesn't.

bool3max 5y ago

PyPI is more complex and probably serves way more traffic.

melissalobos 5y ago

I think the idea here isn't that there isn't a PyPI mirror in Africa. It is that not everyone has internet, so this person wants to have a tiny computer output a local WiFi network people can connect to and download pip packages. Imagine a small town or village with some power and a classroom, but no internet. A teacher could setup a network and have students connect to this device and download packages so they can complete some assignment/make the next Facebook.

bscphil 5y ago
Or that the Internet access they do have is often metered. A friend tells me 100MB costs about 1 USD where he is, which is not an insignificant amount of money. Really puts the whole 300 MB electron app thing in perspective; at any rate it's understandable why having a PyPI mirror in the classroom would be preferable to having each student download the packages over and over.

mjw1007 5y ago

I think PyPI is mostly served via Fastly. https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...

heavyset_go 5y ago

PyPI needs more funding. PyPI even disabled their search API for the pip CLI because of infrastructure overload. It would be nice if more sponsors stepped up to fund their infrastructure.

db65edfc7996 5y ago
I think companies could choose to be more responsible with their usage. Looking at PyPi utilization, I have to imagine the bulk of it comes from CI/CD tooling hammering the servers without any intermediate caching.
- heavyset_go 5y ago
  Definitely. I don't think hammering PyPI's API would be much of problem if the companies doing the hammering paid for the privilege of doing so, though.
- sgtnoodle 5y ago
  That's definitely true. There's so many build jobs running in docker containers that lack a pip cache. I believe my company has its own package servers, but I've specifically avoided adding build system features because of that lack of caching. For one of me, there's probably a dozen people that would just hit the main pypi servers every run.

woodruffw 5y ago

> Mildly surprised a service as big as PyPI doesn't. PyPI uses Fastly as a CDN, which does indeed have presence on every continent (except the big, cold one). The problem here isn't presence, but connectivity to the outside Internet itself. Source: I'm an active developer on PyPI.

qbasic_forever 5y ago

I remember at pycon ~2016 one of the maintainers of pypi did a short talk on it and the entire pypi service at the time only ran on one or two boxes. It was surprisingly scrappy for such a critical service.

tempay 5y ago
This has changed a lot in recent years and is now in a much better state. Their docs give a really nice overview of the evolution over time https://warehouse.pypa.io/application.html https://warehouse.pypa.io/application.html

tentacleuno 5y ago

Serious question: should charities start downloading the entirety of NPM and PyPi (+ Node, NPM, Linux packages, etc.) onto hard drives and passing them onto schools teaching software development in third-world countries alongside the computers they normally send? I'm not sure if that's even possible, or how big the HDD would have to be to store all that, but it would be very very nice nevertheless. I can't imagine not being able to install one-off software packages.