5 ms·
“ Our current CDN “costs” are ~$1.5M/month and not getting smaller. This is generously supported by our CDN provider but is a liability for PyPI’s long-term exi
by Google234 5y ago
“ Our current CDN “costs” are ~$1.5M/month and not getting smaller. This is generously supported by our CDN provider but is a liability for PyPI’s long-term existence.”
Wow
- hirundo 5y agoMaybe distribution via bittorrent is an alternative. It wouldn't be better performance than a good CDN but it distributes the costs better.
- kstrauser 5y ago"Hmm, how can I get people to start using IPFS..." Edit: Not sure about the downvotes. Joking tone aside, that actually seems like an ideal application for it.
- Zababa 5y agoI didn't downvote you but I can attempt to offer an explanation: When reading the first part of your post, I read it as implying that the person you replied to is trying to push a secret agendo or something with IPFS. That's on me of course, and you seem to have no such intentions, but that's how I read it. Also, sarcasm can be hard to understand for some people, and even harder in text form. I think including your "Joking tone aside, that actually seems like an ideal application for it." in your initial comment would have helped. It indicates that what was before was in part a joke, and inform people of your real position.
- kstrauser 5y agoAh, got it. No, I was suggesting that IPFS may be a legitimate option here.
- kortex 5y agoConceptually, I love the idea of bittorrent distribution for binary packages like this. In practice, "oh no, my CI docker builds :(" Too much potential variability. I just want a turnkey way to add multiple hosts/resolvers and use content-addressing. Like bittorrent but with passlists. Is that a thing? Like, IPFS, but I want specific use-case domains. Like I'd be more than willing to stand up some sort of pip proxy for my company's domain but I'd only want it hosting packages used internally.
- toomuchtodo 5y agoNexus? We do this with Nexus.
- namibj 5y agoIf pip had native IPFS support, you could do that by just pinning the packages you use on the local node. You could also have an IPFS gateway with just normal HTTP(S) access control techniques and a normal pip, deferring to the swarm for packages that you haven't pinned. It's unhealthy for the swarm to stop your node from sharing it's data with other nodes, as that'd loose the Torrent swarm effects that unload the initial uploader (PyPI).
- mkj 5y agoExtra variability could be an incentive for CI builds to do their own caching.
- StavrosK 5y agoThere should really exist a transparent pip proxy, then I can set the config of all my CI machines to it and `pip install foo` would do the right thing and install from that cache. I just want a single binary that I can run with zero configuration and have it talk to PyPI and cache packages locally. It would save so much bandwidth.
- rcxdude 5y agoThis is one of the key things which artifactory can do (and it can do it for basically any repository type, not just pypi). It's not so straightforward to set up, however.
- hksh 5y agoI have yet to discover a more-than-default-repository setup that is straightforward when using artifactory.
- Macha 5y agoSuch things certainly exist, we had one in my previous employer though I don't remember if it was something in-house, an artifactory feature, or something open source.
- peterkelly 5y agoIPFS sounds like the ideal solution here. In particular because it would mean you could set up a local node to act as a cache, and pin any packages you rely on.
- viraptor 5y agoI'm eagerly waiting for ipfs-based package distribution channels. They would be great both for local caches and potentially getting closer sources in countries without great mirrors.
- namibj 5y agoTermux (Android app, giving a mostly-normal terminal without root) already uses IPFS. Requiring IPFS for accessing packages/wheels exceeding a threshold from PyPI seems reasonable. Just have PyPI offer collaborative clusters[0] for a few common situations, and maybe work with the IPFS devs to get collaborative clusters to support only pinning a subset by path (like, only pinning the packages you depend on for local CI). [0]: https://cluster.ipfs.io/documentation/collaborative/ https://cluster.ipfs.io/documentation/collaborative/
- StavrosK 5y agoIPFS would be the ideal solution to so many things if it worked well. As it stands, I don't think it would be a solution to anything, it can hardly discover content on other peers in a timeframe less than minutes...
- actually_a_dog 5y agoIf it works for Ubuntu, why not? https://ubuntu.com/download/alternative-downloads https://ubuntu.com/download/alternative-downloads
- dataflow 5y agoBear in mind this isn't just end-users installing on their machines, it also includes continuous integration scripts that run quite frequently.
- westurner 5y ago[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy, "[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c... > Continuous Integration automated build and testing services can help reduce the costs of hosting PyPI by running local mirrors and advising clients in regards to how to efficiently re-build software hundreds or thousands of times a month without re-downloading everything from PyPI every time. [...] > Request from and advisory for CI Services and CI Implementors: > Dear CI Service, > - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds. > - Please advise clients regarding more efficient containerized software build and test strategies. > Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources). Looks like the current figure is significantly higher than $800K/mo for science. How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads: RUN --mount=type=cache,target=/root/.cache/pip RUN --mount=type=cache,target=/home/appuser/.cache/pip
- remram 5y agoFrom an open-source developer's perspective, whether we hit PyPI (a third-party free service) or the cache provided by the CI service (a third-party free service) doesn't seem very different.
- captn3m0 5y agoA 7TB local cache within the CI service is much cheaper to host, since it doesn't have CDN concerns, and bandwidth is much cheaper "within the infra".
- darkr 5y agoThis might be the CDN list price. When you get to higher levels of traffic, you can usually negotiate and commit to a certain level of spend over the next X years, and in exchange the per GB cost falls by at least an order of magnitude. Then again, it might not be - the amount of CI setups out there that download the entire universe every 5 minutes..
- OJFord 5y agoIf it's 'generously covered by the CDN' then I suppose it must be list price, what else would they do? 'We'd probably agree to a 30% discount if you negotiate hard with this load, so that comes to..'?
- H8crilA 5y ago$1.5M/month is about 5-15 software engineers, depending on seniority. Given that this is one of the most popular software repositories of one of the most popular languages it's not actually a lot of money. It isn't cheap but isn't expensive either.
- BugsJustFindMe 5y agoIt sounds like that's a retail price. I wonder what the actual production cost to the CDN is.