5 ms·

Container image distribution seems to be one of the primary problems this tackles: "DevOps ... brings a lot of challenges: the efficiency of image distribution

Container image distribution seems to be one of the primary problems this tackles: "DevOps ... brings a lot of challenges: the efficiency of image distribution, especially when you have a lot of applications and require image distribution at the same time. Dragonfly works extremely well with both Docker and Pouch, and actually we are compatible with any other container technologies without any modifications of container engine." FWIW, this was a similar problem that I tackled for Golang gopher gala hackathon 2015 - a custom bittorrent based docker image registry POC. https://github.com/netvarun/docket https://github.com/netvarun/docket Interestingly my problem statement was somewhat similar: "Large scale deploys are going to choke your docker registry. Imagine pulling/deploying a 800mb base image (for example the official perl image) across 200 machines in one go. That's 800*200 = 160GB [EDIT: Correction thanks to kingbirdy] of data that's going to be deployed and it'll definitely choke your private docker registry (and will take a while pulling in from the public image)."

kingbirdy 8y ago
It would be 160GB, not 1.6TB
- netvarun 8y ago
  Thanks, fixed.
ithkuil 8y ago
any thoughts on https://github.com/jvassev/image2ipfs/ https://github.com/jvassev/image2ipfs/ ?
oscargrouch 8y ago
Devs need to be aware that bittorrent now(for sometime) have a DHT solution that allows to have "mutable" slots with the crypto public address tied up to it. So if you own the private key you can write the payload, and just share your public-key address to the people you want to share the payload with. In the payload you can write a traditional immutable torrent manifest for instance, which is in essence a public-key crypto based update system. For a lot of cases i think it's a better approach than what IPFS and DAT provides, because you dont care about it being a global address. All you want is to share with a group of people, more in the p2p social/organic way. I was playing with it once, using the libtorrent library and the main bittorrent DHT, and it was a very nice experience.. it finds the payload pretty fast when you think its a DHT, and you are working in pure p2p fashion. The only single point of failure here is the DHT bootstrap peer. Im planing to use this feature to distribute binary images for clients that have my public key.
- voltagex_ 8y ago
  Is there anything I can play with for mutable torrents? I thought it was still only in research paper form.
  - namibj 8y ago
    libtorrent has support for it [0], as far as I know. The "research paper" you probably mean [1] is the torrent equivalent of and RFC. AFAIK at least python uses something very similar. There is an apparently Node.js implementation [2] of something that can publish a given torrent to a mutable address, and also retrieve a mutable torrent from a given address. I do not know how well this is implemented in the usual clients, but if you want it in your docker, you might want to talk to libtorrent directly, and implementing BEP 46 yourself should not be hard with the things the library has to offer. A benefit could be that depending on how you handle it, you might be able to store the tarballs docker images seem to be in their unpacked form, and just keep some metadata about what the header(s) of the tarball were, along with some file offsets. This way you would be relieved of the unnecessary storage burden, and able to possibly use many more of your servers to seed at least part of the images, e.g. maybe only the parts that are not mutated when the software is running. E.g., download once, unpack, only offer to seed those files/pieces that did not get modified in the meantime, without trying to re-download the "broken" data. [0]: https://www.libtorrent.org/dht_store.html https://www.libtorrent.org/dht_store.html [1]: http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html [2]: https://github.com/lmatteis/dmt https://github.com/lmatteis/dmt
- brokenmachine 8y ago
  Very interesting. Forgive my ignorance, but you mean it's kind of like mutable torrents that only the uploader can modify? Does a client just search the DHT for the public key? I thought torrent clients searched for the hash of the files. If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? Are there any example projects using this?

kingbirdy 8y ago

It would be 160GB, not 1.6TB

netvarun 8y ago
Thanks, fixed.

ithkuil 8y ago

any thoughts on https://github.com/jvassev/image2ipfs/ https://github.com/jvassev/image2ipfs/ ?

oscargrouch 8y ago

Devs need to be aware that bittorrent now(for sometime) have a DHT solution that allows to have "mutable" slots with the crypto public address tied up to it. So if you own the private key you can write the payload, and just share your public-key address to the people you want to share the payload with. In the payload you can write a traditional immutable torrent manifest for instance, which is in essence a public-key crypto based update system. For a lot of cases i think it's a better approach than what IPFS and DAT provides, because you dont care about it being a global address. All you want is to share with a group of people, more in the p2p social/organic way. I was playing with it once, using the libtorrent library and the main bittorrent DHT, and it was a very nice experience.. it finds the payload pretty fast when you think its a DHT, and you are working in pure p2p fashion. The only single point of failure here is the DHT bootstrap peer. Im planing to use this feature to distribute binary images for clients that have my public key.

voltagex_ 8y ago
Is there anything I can play with for mutable torrents? I thought it was still only in research paper form.
- namibj 8y ago
  libtorrent has support for it [0], as far as I know. The "research paper" you probably mean [1] is the torrent equivalent of and RFC. AFAIK at least python uses something very similar. There is an apparently Node.js implementation [2] of something that can publish a given torrent to a mutable address, and also retrieve a mutable torrent from a given address. I do not know how well this is implemented in the usual clients, but if you want it in your docker, you might want to talk to libtorrent directly, and implementing BEP 46 yourself should not be hard with the things the library has to offer. A benefit could be that depending on how you handle it, you might be able to store the tarballs docker images seem to be in their unpacked form, and just keep some metadata about what the header(s) of the tarball were, along with some file offsets. This way you would be relieved of the unnecessary storage burden, and able to possibly use many more of your servers to seed at least part of the images, e.g. maybe only the parts that are not mutated when the software is running. E.g., download once, unpack, only offer to seed those files/pieces that did not get modified in the meantime, without trying to re-download the "broken" data. [0]: https://www.libtorrent.org/dht_store.html https://www.libtorrent.org/dht_store.html [1]: http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html [2]: https://github.com/lmatteis/dmt https://github.com/lmatteis/dmt
brokenmachine 8y ago
Very interesting. Forgive my ignorance, but you mean it's kind of like mutable torrents that only the uploader can modify? Does a client just search the DHT for the public key? I thought torrent clients searched for the hash of the files. If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? Are there any example projects using this?