5 ms·

Conceptually, I love the idea of bittorrent distribution for binary packages like this. In practice, "oh no, my CI docker builds :(" Too much potential variabil

by kortex 5y ago

Conceptually, I love the idea of bittorrent distribution for binary packages like this. In practice, "oh no, my CI docker builds :(" Too much potential variability. I just want a turnkey way to add multiple hosts/resolvers and use content-addressing. Like bittorrent but with passlists. Is that a thing? Like, IPFS, but I want specific use-case domains. Like I'd be more than willing to stand up some sort of pip proxy for my company's domain but I'd only want it hosting packages used internally.

toomuchtodo 5y ago
Nexus? We do this with Nexus.
namibj 5y ago
If pip had native IPFS support, you could do that by just pinning the packages you use on the local node. You could also have an IPFS gateway with just normal HTTP(S) access control techniques and a normal pip, deferring to the swarm for packages that you haven't pinned. It's unhealthy for the swarm to stop your node from sharing it's data with other nodes, as that'd loose the Torrent swarm effects that unload the initial uploader (PyPI).
mkj 5y ago
Extra variability could be an incentive for CI builds to do their own caching.
StavrosK 5y ago
There should really exist a transparent pip proxy, then I can set the config of all my CI machines to it and `pip install foo` would do the right thing and install from that cache. I just want a single binary that I can run with zero configuration and have it talk to PyPI and cache packages locally. It would save so much bandwidth.
- rcxdude 5y ago
  This is one of the key things which artifactory can do (and it can do it for basically any repository type, not just pypi). It's not so straightforward to set up, however.
  - hksh 5y ago
    I have yet to discover a more-than-default-repository setup that is straightforward when using artifactory.
- Macha 5y ago
  Such things certainly exist, we had one in my previous employer though I don't remember if it was something in-house, an artifactory feature, or something open source.
  - StavrosK 5y ago
    Hmm, if you were using Artifactory, it sounds like that's what was doing that. I'm not aware of a dead simple, plug-and-play thing, though.
- Epic_Wink 5y ago
  Most CI vendors offer caching directories between builds, and then you set the pip cache directory. Failing that, I made a proxy server that can work transparently if you can set default environment variables: check out proxpi
  - StavrosK 5y ago
    proxpi is exactly what I was looking for, thank you! The straightforward Docker installation is especially great, I can have it running with one command.
- ArchOversight 5y ago
  You are in luck, such a tool exists: https://www.devpi.net https://www.devpi.net In the past I've run this in a docker container, setup some pip environment variables and its off to the races. It transparently caches stuff from PyPi and keeps things local.

by kortex 5y ago

toomuchtodo 5y ago

Nexus? We do this with Nexus.

namibj 5y ago

If pip had native IPFS support, you could do that by just pinning the packages you use on the local node. You could also have an IPFS gateway with just normal HTTP(S) access control techniques and a normal pip, deferring to the swarm for packages that you haven't pinned. It's unhealthy for the swarm to stop your node from sharing it's data with other nodes, as that'd loose the Torrent swarm effects that unload the initial uploader (PyPI).

mkj 5y ago

Extra variability could be an incentive for CI builds to do their own caching.

StavrosK 5y ago

There should really exist a transparent pip proxy, then I can set the config of all my CI machines to it and `pip install foo` would do the right thing and install from that cache. I just want a single binary that I can run with zero configuration and have it talk to PyPI and cache packages locally. It would save so much bandwidth.

rcxdude 5y ago
This is one of the key things which artifactory can do (and it can do it for basically any repository type, not just pypi). It's not so straightforward to set up, however.
- hksh 5y ago
  I have yet to discover a more-than-default-repository setup that is straightforward when using artifactory.
Macha 5y ago
Such things certainly exist, we had one in my previous employer though I don't remember if it was something in-house, an artifactory feature, or something open source.
- StavrosK 5y ago
  Hmm, if you were using Artifactory, it sounds like that's what was doing that. I'm not aware of a dead simple, plug-and-play thing, though.
Epic_Wink 5y ago
Most CI vendors offer caching directories between builds, and then you set the pip cache directory. Failing that, I made a proxy server that can work transparently if you can set default environment variables: check out proxpi
- StavrosK 5y ago
  proxpi is exactly what I was looking for, thank you! The straightforward Docker installation is especially great, I can have it running with one command.
ArchOversight 5y ago
You are in luck, such a tool exists: https://www.devpi.net https://www.devpi.net In the past I've run this in a docker container, setup some pip environment variables and its off to the races. It transparently caches stuff from PyPi and keeps things local.