13 ms·

Maybe distribution via bittorrent is an alternative. It wouldn't be better performance than a good CDN but it distributes the costs better.

kstrauser 5y ago
"Hmm, how can I get people to start using IPFS..." Edit: Not sure about the downvotes. Joking tone aside, that actually seems like an ideal application for it.
- Zababa 5y ago
  I didn't downvote you but I can attempt to offer an explanation: When reading the first part of your post, I read it as implying that the person you replied to is trying to push a secret agendo or something with IPFS. That's on me of course, and you seem to have no such intentions, but that's how I read it. Also, sarcasm can be hard to understand for some people, and even harder in text form. I think including your "Joking tone aside, that actually seems like an ideal application for it." in your initial comment would have helped. It indicates that what was before was in part a joke, and inform people of your real position.
  - kstrauser 5y ago
    Ah, got it. No, I was suggesting that IPFS may be a legitimate option here.
kortex 5y ago
Conceptually, I love the idea of bittorrent distribution for binary packages like this. In practice, "oh no, my CI docker builds :(" Too much potential variability. I just want a turnkey way to add multiple hosts/resolvers and use content-addressing. Like bittorrent but with passlists. Is that a thing? Like, IPFS, but I want specific use-case domains. Like I'd be more than willing to stand up some sort of pip proxy for my company's domain but I'd only want it hosting packages used internally.
- toomuchtodo 5y ago
  Nexus? We do this with Nexus.
- namibj 5y ago
  If pip had native IPFS support, you could do that by just pinning the packages you use on the local node. You could also have an IPFS gateway with just normal HTTP(S) access control techniques and a normal pip, deferring to the swarm for packages that you haven't pinned. It's unhealthy for the swarm to stop your node from sharing it's data with other nodes, as that'd loose the Torrent swarm effects that unload the initial uploader (PyPI).
- mkj 5y ago
  Extra variability could be an incentive for CI builds to do their own caching.
- StavrosK 5y ago
  There should really exist a transparent pip proxy, then I can set the config of all my CI machines to it and `pip install foo` would do the right thing and install from that cache. I just want a single binary that I can run with zero configuration and have it talk to PyPI and cache packages locally. It would save so much bandwidth.
  - rcxdude 5y ago
    This is one of the key things which artifactory can do (and it can do it for basically any repository type, not just pypi). It's not so straightforward to set up, however.
    - hksh 5y ago
      I have yet to discover a more-than-default-repository setup that is straightforward when using artifactory.
  - Macha 5y ago
    Such things certainly exist, we had one in my previous employer though I don't remember if it was something in-house, an artifactory feature, or something open source.
peterkelly 5y ago
IPFS sounds like the ideal solution here. In particular because it would mean you could set up a local node to act as a cache, and pin any packages you rely on.
- viraptor 5y ago
  I'm eagerly waiting for ipfs-based package distribution channels. They would be great both for local caches and potentially getting closer sources in countries without great mirrors.
- namibj 5y ago
  Termux (Android app, giving a mostly-normal terminal without root) already uses IPFS. Requiring IPFS for accessing packages/wheels exceeding a threshold from PyPI seems reasonable. Just have PyPI offer collaborative clusters[0] for a few common situations, and maybe work with the IPFS devs to get collaborative clusters to support only pinning a subset by path (like, only pinning the packages you depend on for local CI). [0]: https://cluster.ipfs.io/documentation/collaborative/ https://cluster.ipfs.io/documentation/collaborative/
- StavrosK 5y ago
  IPFS would be the ideal solution to so many things if it worked well. As it stands, I don't think it would be a solution to anything, it can hardly discover content on other peers in a timeframe less than minutes...
actually_a_dog 5y ago
If it works for Ubuntu, why not? https://ubuntu.com/download/alternative-downloads https://ubuntu.com/download/alternative-downloads

kstrauser 5y ago

"Hmm, how can I get people to start using IPFS..." Edit: Not sure about the downvotes. Joking tone aside, that actually seems like an ideal application for it.

Zababa 5y ago
I didn't downvote you but I can attempt to offer an explanation: When reading the first part of your post, I read it as implying that the person you replied to is trying to push a secret agendo or something with IPFS. That's on me of course, and you seem to have no such intentions, but that's how I read it. Also, sarcasm can be hard to understand for some people, and even harder in text form. I think including your "Joking tone aside, that actually seems like an ideal application for it." in your initial comment would have helped. It indicates that what was before was in part a joke, and inform people of your real position.
- kstrauser 5y ago
  Ah, got it. No, I was suggesting that IPFS may be a legitimate option here.

kortex 5y ago

Conceptually, I love the idea of bittorrent distribution for binary packages like this. In practice, "oh no, my CI docker builds :(" Too much potential variability. I just want a turnkey way to add multiple hosts/resolvers and use content-addressing. Like bittorrent but with passlists. Is that a thing? Like, IPFS, but I want specific use-case domains. Like I'd be more than willing to stand up some sort of pip proxy for my company's domain but I'd only want it hosting packages used internally.

toomuchtodo 5y ago
Nexus? We do this with Nexus.
namibj 5y ago
If pip had native IPFS support, you could do that by just pinning the packages you use on the local node. You could also have an IPFS gateway with just normal HTTP(S) access control techniques and a normal pip, deferring to the swarm for packages that you haven't pinned. It's unhealthy for the swarm to stop your node from sharing it's data with other nodes, as that'd loose the Torrent swarm effects that unload the initial uploader (PyPI).
mkj 5y ago
Extra variability could be an incentive for CI builds to do their own caching.
StavrosK 5y ago
There should really exist a transparent pip proxy, then I can set the config of all my CI machines to it and `pip install foo` would do the right thing and install from that cache. I just want a single binary that I can run with zero configuration and have it talk to PyPI and cache packages locally. It would save so much bandwidth.
- rcxdude 5y ago
  This is one of the key things which artifactory can do (and it can do it for basically any repository type, not just pypi). It's not so straightforward to set up, however.
  - hksh 5y ago
    I have yet to discover a more-than-default-repository setup that is straightforward when using artifactory.
- Macha 5y ago
  Such things certainly exist, we had one in my previous employer though I don't remember if it was something in-house, an artifactory feature, or something open source.

peterkelly 5y ago

IPFS sounds like the ideal solution here. In particular because it would mean you could set up a local node to act as a cache, and pin any packages you rely on.

viraptor 5y ago
I'm eagerly waiting for ipfs-based package distribution channels. They would be great both for local caches and potentially getting closer sources in countries without great mirrors.
namibj 5y ago
Termux (Android app, giving a mostly-normal terminal without root) already uses IPFS. Requiring IPFS for accessing packages/wheels exceeding a threshold from PyPI seems reasonable. Just have PyPI offer collaborative clusters[0] for a few common situations, and maybe work with the IPFS devs to get collaborative clusters to support only pinning a subset by path (like, only pinning the packages you depend on for local CI). [0]: https://cluster.ipfs.io/documentation/collaborative/ https://cluster.ipfs.io/documentation/collaborative/
StavrosK 5y ago
IPFS would be the ideal solution to so many things if it worked well. As it stands, I don't think it would be a solution to anything, it can hardly discover content on other peers in a timeframe less than minutes...

actually_a_dog 5y ago

If it works for Ubuntu, why not? https://ubuntu.com/download/alternative-downloads https://ubuntu.com/download/alternative-downloads