5 ms·

It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol

It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters. https://github.com/uber/kraken https://github.com/uber/kraken

XorNot 3y ago
The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).
- infogulch 3y ago
  Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs
apichat 3y ago
https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."
- anacrolix 3y ago
  That's exactly the problem my BitTorrent client solves. https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent. It's been around since 2013.
phillebaba 3y ago
Kraken is sadly a dead project, with little work being done. For example support for Containerd is non-existent or just not documented. I created Spegel to fill the gap but focus on the P2P registry component without the overhead of running a stateful application. https://github.com/spegel-org/spegel https://github.com/spegel-org/spegel

XorNot 3y ago

The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).

infogulch 3y ago
Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs

apichat 3y ago

https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."

anacrolix 3y ago
That's exactly the problem my BitTorrent client solves. https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent. It's been around since 2013.

phillebaba 3y ago

Kraken is sadly a dead project, with little work being done. For example support for Containerd is non-existent or just not documented. I created Spegel to fill the gap but focus on the P2P registry component without the overhead of running a stateful application. https://github.com/spegel-org/spegel https://github.com/spegel-org/spegel