4 ms·

PyPI in a Box (2020)

88 points by luu 5y ago

woodruffw 5y ago
This is great work! Python packaging is complicated for many reasons (both good and bad), but PyPI's index format is delightfully simple. Projects like this reinforce my opinion that keeping it simple has been a great decision by the Python community and PyPI admins.
- jborean93 5y ago
  In don’t wish to take away from pypiserver and how nice and simple it is but I found it’s missing certain features that are present in the PyPI Wharehouse that underpins pypi.org. If I remember it correctly it didn’t expose the python_requires metadata or maybe something to do with extras that to wanted to test. It was definitely an edge case but still something to be aware off.
- dralley 5y ago
  >Python packaging is complicated for many reasons (both good and bad), but PyPI's index format is delightfully simple. In a lot of the worst ways. Dependency resolution can't be done locally, it requires tons of separate network requests to fetch all the requirements.
  - danmur 5y ago
    I don't have as many complaints about Python's dependency system as most people but that issue is very painful. Calculating dependencies properly is slow and very vulnerable to network failures, causes constant issues inside corporate networks with crappy VPNs and unreliable networks.
  - woodruffw 5y ago
    Yeah, this is probably the single largest wart in the Python packaging ecosystem. It's gotten better over time (wheels improve things, as do third-party tools like Pipenv and pip-compile), but there's still a lot to do.
digisign 5y ago
We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough Mildly surprised a service as big as PyPI doesn't.
- bool3max 5y ago
  PyPI is more complex and probably serves way more traffic.
- melissalobos 5y ago
  I think the idea here isn't that there isn't a PyPI mirror in Africa. It is that not everyone has internet, so this person wants to have a tiny computer output a local WiFi network people can connect to and download pip packages. Imagine a small town or village with some power and a classroom, but no internet. A teacher could setup a network and have students connect to this device and download packages so they can complete some assignment/make the next Facebook.
  - bscphil 5y ago
    Or that the Internet access they do have is often metered. A friend tells me 100MB costs about 1 USD where he is, which is not an insignificant amount of money. Really puts the whole 300 MB electron app thing in perspective; at any rate it's understandable why having a PyPI mirror in the classroom would be preferable to having each student download the packages over and over.
- mjw1007 5y ago
  I think PyPI is mostly served via Fastly. https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...
- heavyset_go 5y ago
  PyPI needs more funding. PyPI even disabled their search API for the pip CLI because of infrastructure overload. It would be nice if more sponsors stepped up to fund their infrastructure.
dralley 5y ago
You can also do this with Pulp, and have it act as a caching proxy that lazily caches the packages only when they first get downloaded. It's a lot more heavyweight though, so maybe it's not the best choice for a Raspberry Pi.
quietbritishjim 5y ago
devpi acts as a caching proxy for PyPI and takes a bit less setup than this. Plus, you can use it for storing your own packages in a separate index. https://github.com/devpi/devpi https://github.com/devpi/devpi
simonw 5y ago
The developer experience in regions that don't have fast internet access is hard to imagine, especially with bandwidth-hogs like the npm ecosystem. See also https://meyerweb.com/eric/thoughts/2018/08/07/securing-sites-made-them-less-accessible/ https://meyerweb.com/eric/thoughts/2018/08/07/securing-sites... - which points out that when every site moved to https it broke local caching proxies, which had a big negative impact on people in countries with slower internet.
- tentacleuno 5y ago
  Regarding the article you sent: > Beyond deploying service workers and hoping those struggling to bridge the digital divide make it across, I don’t really have a solution here. One way to get around this would be a MITM HTTPS certificate, which would allow a local caching proxy to decrypt and encrypt requests. This means you'd have to install the certificate on each device you wish to be compatible with the caching proxy (otherwise it can't decrypt it), though.
yjftsjthsd-h 5y ago
> At the time of writing this post, the entire PyPI repository is somewhere in the neighbourhood of 1TB but, by using a selective download, I was able to get it down to 120GB or so. That's the surprising part to me; that config doesn't look like it's filtering by popularity or such, just python versions? I guess packages aren't huge, but I'm still surprised that it's so small.
- woodruffw 5y ago
  The average Python package distribution is pretty small: source distributions are (or should be) compressed source code, and binary distributions ("wheels") should be (mostly) compressed Python bytecode and the occasional platform-specific binary. There are also relatively conservative per-project and per-releaase upload limits. PyPI's admins can increase those limits on a project or individual release, but the baseline limits have seemingly preserved the commons reasonably well.
- oefrha 5y ago
  Maybe PyPI was that small when the post was written. It currently sits at 10.4TB; TensorFlow alone is more than 1TB. I suppose you can still keep the mirror relatively small if you exclude most of the giant ML wheels. https://pypi.org/stats/ https://pypi.org/stats/
mistrial9 5y ago
Q. where is the last version 2.7 PyPi archived?
- qbasic_forever 5y ago
  You mean python? https://www.python.org/downloads/ https://www.python.org/downloads/ still has 2.7 If you're looking for libraries that support 2.7 you're in for a lot more work, you'd have to go library by library and figure out when was the last version they produced that supported 2.7. Almost certainly it will be way out of date and depend on other out of date things. There was no single moment where every library 'threw the switch' and moved off 2.7 to 3.x, it happened organically over about 10 years. A lot of libraries spent a significant amount of time (and code) supporting both 2.7 and 3.x from the same codebase.
animal_spirits 5y ago
I bet this can be added to the Internet In a Box project. It already comes with python installed. https://internet-in-a-box.org/ https://internet-in-a-box.org/
deleted 5y ago
[deleted]

88 points by luu 5y ago

This is great work! Python packaging is complicated for many reasons (both good and bad), but PyPI's index format is delightfully simple. Projects like this reinforce my opinion that keeping it simple has been a great decision by the Python community and PyPI admins.

jborean93 5y ago
In don’t wish to take away from pypiserver and how nice and simple it is but I found it’s missing certain features that are present in the PyPI Wharehouse that underpins pypi.org. If I remember it correctly it didn’t expose the python_requires metadata or maybe something to do with extras that to wanted to test. It was definitely an edge case but still something to be aware off.
dralley 5y ago
>Python packaging is complicated for many reasons (both good and bad), but PyPI's index format is delightfully simple. In a lot of the worst ways. Dependency resolution can't be done locally, it requires tons of separate network requests to fetch all the requirements.
- danmur 5y ago
  I don't have as many complaints about Python's dependency system as most people but that issue is very painful. Calculating dependencies properly is slow and very vulnerable to network failures, causes constant issues inside corporate networks with crappy VPNs and unreliable networks.
- woodruffw 5y ago
  Yeah, this is probably the single largest wart in the Python packaging ecosystem. It's gotten better over time (wheels improve things, as do third-party tools like Pipenv and pip-compile), but there's still a lot to do.

digisign 5y ago

We had mirrors on every continent for linux distributions (and other things) way back before the turn of the century! cough cough Mildly surprised a service as big as PyPI doesn't.

bool3max 5y ago
PyPI is more complex and probably serves way more traffic.
melissalobos 5y ago
I think the idea here isn't that there isn't a PyPI mirror in Africa. It is that not everyone has internet, so this person wants to have a tiny computer output a local WiFi network people can connect to and download pip packages. Imagine a small town or village with some power and a classroom, but no internet. A teacher could setup a network and have students connect to this device and download packages so they can complete some assignment/make the next Facebook.
- bscphil 5y ago
  Or that the Internet access they do have is often metered. A friend tells me 100MB costs about 1 USD where he is, which is not an insignificant amount of money. Really puts the whole 300 MB electron app thing in perspective; at any rate it's understandable why having a PyPI mirror in the classroom would be preferable to having each student download the packages over and over.
mjw1007 5y ago
I think PyPI is mostly served via Fastly. https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...
heavyset_go 5y ago
PyPI needs more funding. PyPI even disabled their search API for the pip CLI because of infrastructure overload. It would be nice if more sponsors stepped up to fund their infrastructure.

dralley 5y ago

You can also do this with Pulp, and have it act as a caching proxy that lazily caches the packages only when they first get downloaded. It's a lot more heavyweight though, so maybe it's not the best choice for a Raspberry Pi.

quietbritishjim 5y ago

devpi acts as a caching proxy for PyPI and takes a bit less setup than this. Plus, you can use it for storing your own packages in a separate index. https://github.com/devpi/devpi https://github.com/devpi/devpi

simonw 5y ago

The developer experience in regions that don't have fast internet access is hard to imagine, especially with bandwidth-hogs like the npm ecosystem. See also https://meyerweb.com/eric/thoughts/2018/08/07/securing-sites-made-them-less-accessible/ https://meyerweb.com/eric/thoughts/2018/08/07/securing-sites... - which points out that when every site moved to https it broke local caching proxies, which had a big negative impact on people in countries with slower internet.

tentacleuno 5y ago
Regarding the article you sent: > Beyond deploying service workers and hoping those struggling to bridge the digital divide make it across, I don’t really have a solution here. One way to get around this would be a MITM HTTPS certificate, which would allow a local caching proxy to decrypt and encrypt requests. This means you'd have to install the certificate on each device you wish to be compatible with the caching proxy (otherwise it can't decrypt it), though.

yjftsjthsd-h 5y ago

> At the time of writing this post, the entire PyPI repository is somewhere in the neighbourhood of 1TB but, by using a selective download, I was able to get it down to 120GB or so. That's the surprising part to me; that config doesn't look like it's filtering by popularity or such, just python versions? I guess packages aren't huge, but I'm still surprised that it's so small.

woodruffw 5y ago
The average Python package distribution is pretty small: source distributions are (or should be) compressed source code, and binary distributions ("wheels") should be (mostly) compressed Python bytecode and the occasional platform-specific binary. There are also relatively conservative per-project and per-releaase upload limits. PyPI's admins can increase those limits on a project or individual release, but the baseline limits have seemingly preserved the commons reasonably well.
oefrha 5y ago
Maybe PyPI was that small when the post was written. It currently sits at 10.4TB; TensorFlow alone is more than 1TB. I suppose you can still keep the mirror relatively small if you exclude most of the giant ML wheels. https://pypi.org/stats/ https://pypi.org/stats/

mistrial9 5y ago

Q. where is the last version 2.7 PyPi archived?

qbasic_forever 5y ago
You mean python? https://www.python.org/downloads/ https://www.python.org/downloads/ still has 2.7 If you're looking for libraries that support 2.7 you're in for a lot more work, you'd have to go library by library and figure out when was the last version they produced that supported 2.7. Almost certainly it will be way out of date and depend on other out of date things. There was no single moment where every library 'threw the switch' and moved off 2.7 to 3.x, it happened organically over about 10 years. A lot of libraries spent a significant amount of time (and code) supporting both 2.7 and 3.x from the same codebase.

animal_spirits 5y ago

I bet this can be added to the Internet In a Box project. It already comes with python installed. https://internet-in-a-box.org/ https://internet-in-a-box.org/

deleted 5y ago

[deleted]