4 ms·

> "Distributed cloud backup" Relatedly, when Docker started charging open source projects for Dockerhub storage, a friend and I started brainstorming a torrent

> "Distributed cloud backup" Relatedly, when Docker started charging open source projects for Dockerhub storage, a friend and I started brainstorming a torrent registry driver (although we didn't find a way around the project still needing to provide a signature for verification at least, even if it didn't need to maintain hosting for the image itself) Nothing to show for it just yet, but point being I think this "distributed cloud backup" idea has a lot of interesting applications still.

londons_explore 3y ago
What's the signature problem? For a read-only registry, signatures can all be included in the torrent. For a versioned registry, all the info can be stored in any cloud key-value store - eg. The bittorrent DHT
- indigochill 3y ago
  It could just be a lack of understanding on our part, but here's the scenario we're thinking about: Say the Redis team seeds a redis:1.0 image. So far, so good. But what if a nefarious hacker seeds a redis:2.0 image? In a traditional registry, the auth system for the registry ensures that only those who are authorized can write to a particular registry. But in (at least our understanding of) a swarm, there is no auth and therefore no way to know if a particular tag of the redis image in the swarm is legitimate or not. if I understand you, this is where you're saying the info can be stored in the DHT, which I don't really understand at the moment but if so that sounds like something we should look into.
  - rakoo 3y ago
    It has always been possible to store immutable data in the DHT, that's partly what it was built for. "This is a sha1 of a content, I want to know what content is behind" There is a BEP (Bittorrent Extension Protocol) for storing mutable data: like all good things in computing, it's just an indirection. Create a pair of asymetrical cryptographic keys for a specific (mutable) content and store the hash of the public key. People will search for that hash, get to some content that is any bytes you want but signed with the private key. What you want is not exactly that, but more of a naming system. The easiest would be to let each image owner be their own source of truth: the redis team puts up an HTTPS server with a well known path, say "/.well-known/docker?identity=redis:2.0" and the reply includes the magnet link. My node would cache the association under the assumption that it never changes (it shouldn't; a change in the image implies a change in the name). As you yourselves discovered, the swarm doesn't solve the naming issue (ie Zooko's triangle), there needs to be a third-party here
    - anacrolix 3y ago
      https://github.com/anacrolix/btlink https://github.com/anacrolix/btlink
didntcheck 3y ago
It sounds like IPFS (or other content-addressed distributed data stores) may be more what you're after there than Bittorrent
- indigochill 3y ago
  IPFS would be another potential solution. What drew us to Bittorrent for this use case is the (at least original) ethos of clients both downloading and uploading cooperatively such that as long as there's any community that values the image, it will continue to be available (which would basically be true for an IPFS implementation as well). But the registry implementation in the "distribution" project takes storage drivers, so an IPFS storage driver is certainly a possibility. In part for me going with Bittorrent was also just an excuse to get more familiar with the technical details of Bittorrent.

londons_explore 3y ago

What's the signature problem? For a read-only registry, signatures can all be included in the torrent. For a versioned registry, all the info can be stored in any cloud key-value store - eg. The bittorrent DHT

indigochill 3y ago
It could just be a lack of understanding on our part, but here's the scenario we're thinking about: Say the Redis team seeds a redis:1.0 image. So far, so good. But what if a nefarious hacker seeds a redis:2.0 image? In a traditional registry, the auth system for the registry ensures that only those who are authorized can write to a particular registry. But in (at least our understanding of) a swarm, there is no auth and therefore no way to know if a particular tag of the redis image in the swarm is legitimate or not. if I understand you, this is where you're saying the info can be stored in the DHT, which I don't really understand at the moment but if so that sounds like something we should look into.
- rakoo 3y ago
  It has always been possible to store immutable data in the DHT, that's partly what it was built for. "This is a sha1 of a content, I want to know what content is behind" There is a BEP (Bittorrent Extension Protocol) for storing mutable data: like all good things in computing, it's just an indirection. Create a pair of asymetrical cryptographic keys for a specific (mutable) content and store the hash of the public key. People will search for that hash, get to some content that is any bytes you want but signed with the private key. What you want is not exactly that, but more of a naming system. The easiest would be to let each image owner be their own source of truth: the redis team puts up an HTTPS server with a well known path, say "/.well-known/docker?identity=redis:2.0" and the reply includes the magnet link. My node would cache the association under the assumption that it never changes (it shouldn't; a change in the image implies a change in the name). As you yourselves discovered, the swarm doesn't solve the naming issue (ie Zooko's triangle), there needs to be a third-party here
  - anacrolix 3y ago
    https://github.com/anacrolix/btlink https://github.com/anacrolix/btlink

didntcheck 3y ago

It sounds like IPFS (or other content-addressed distributed data stores) may be more what you're after there than Bittorrent

indigochill 3y ago
IPFS would be another potential solution. What drew us to Bittorrent for this use case is the (at least original) ethos of clients both downloading and uploading cooperatively such that as long as there's any community that values the image, it will continue to be available (which would basically be true for an IPFS implementation as well). But the registry implementation in the "distribution" project takes storage drivers, so an IPFS storage driver is certainly a possibility. In part for me going with Bittorrent was also just an excuse to get more familiar with the technical details of Bittorrent.