4 ms·

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well downloa

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

saagarjha 6y ago
> Most of us download our OS via torrents only I mean, I only do it to stick it to the people who claim torrents can only be used for piracy; I think most people prefer the simplicity of direct downloads though…
- bigmanwalter 6y ago
  I can usually max out my connection speed with torrents. This is rarely the case with direct downloads.
  - saagarjha 6y ago
    Guess I just have slow internet :(
  - stingraycharles 6y ago
    The start up time of a torrent is typically so long though, by the time it has connected to all peers and is downloading I have already downloaded the iso with a direct download anyway.
    - stonewareslord 6y ago
      There's a feature of linx-server that provides uploaded files with a torrent URL as well as a regular download URL. I believe the torrent client tries fetching data from peers as well as from linx (http). https://github.com/andreimarcu/linx-server https://github.com/andreimarcu/linx-server
    - namibj 6y ago
      Soon, when QUIC is widely supported by the swarm (just needs updating to the current version), IPFS should be better than torrents for start-up delay, and much more importantly, it allows for sharing data across "torrents" when using content-defined chunking via rabin or buzhash. This means that things like common larger binaries get shared between images that include them, which should greatly increase the average amount of seeders for the chunks that make up an image.
  - skunkworker 6y ago
    On symmetric gigabit fiber only a few services, mainly steam and battle.net have ever given me 90mbyte/s download speeds. A surprising amount will limit to 500mbit or less, even if you have the download pipe for it.
- Crazyontap 6y ago
  At least for Ubuntu and popular distros like mint you get super high speeds on torrent maybe because you have a peer in the same region. Afaik Windows is also using this to install updates where it shares the download with others in the region (1) using p2p (though i may be wrong since i don't use Windows anymore) (1) https://www.itproportal.com/amp/news/how-to-stop-windows-10-sharing-updates-with-your-internet-connection/ https://www.itproportal.com/amp/news/how-to-stop-windows-10-...
- Nullabillity 6y ago
  Docker images already need special handling since you download the layers separately and reassemble them. Going from that to full BitTorrent should be transparent to the users. In fact, there already exist several implementations of it for Docker![0, 1, 2] [0]: https://coreos.com/blog/torrent-pulls https://coreos.com/blog/torrent-pulls [1]: https://d7y.io/en-us/ https://d7y.io/en-us/ [2]: https://github.com/uber/kraken https://github.com/uber/kraken
bsamuels 6y ago
The hardest part of that would be verifying image authenticity. Google Cloud uses an adjacent feature called binary authorization. When turned on, only images that are signed by a given authority (usually your ci/cd instruments) can be run inside your Kubernetes cluster. Binary authorization may be a good starting point for someone trying to make bittorrent distributed images a usable thing.
- sneak 6y ago
  > The hardest part of that would be verifying image authenticity. That's exactly what Bittorrent does with its hash tree. You'd get the root hash (extremely tiny) from Docker Hub, and the rest of the metadata, as well as the data blocks, from the swarm. The authenticity is all handled by the TLS that serves you the root infohash from Docker Hub. It's a Merkle tree: the root hash is for the metadata, which is a list of hashes of the blocks.
  - im3w1l 6y ago
    If this reads like greek, let me dumb it down. You get the hash of the final result from the trusted server and hash is checked. Because of this you will never get an invalid image. There are also some clever tricks to make sure no one can force you to start over from scratch by sending wrong data. But that's more of a detail.
- pixl97 6y ago
  Why, the website could still have the hash for the image which is only a few KB versus hundreds of megabytes or even gigabytes. Just have the bit-docker app check the hash before executing.
- jtsiskin 6y ago
  What is the issue here? Is the torrent checksum (provided by docker) not enough?
  - bsamuels 6y ago
    Let's say someone hacks a maintainer for the Ubuntu base image. The hacker publishes a new version of the base image with a backdoor. When the backdoor is detected, you now need a revocation system so the distribution of the malicious image will die. You can theoretically do this on the tracker level, but people may build other trackers that may not propagate the changes.
user5994461 6y ago
How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy.
- Crazyontap 6y ago
  - Its an alternative download solution - There is no rate limiting for paid accounts or companies from what I see in the article.
- znpy 6y ago
  The company proxy would have to be modified to allow torrent traffic, I guess?
- cogman10 6y ago
  You could have a mixed mode pretty easily. Heck, it would even be pretty efficient. For your own company, you'd host a swarm that is firewalled in. Then when someone says "I want image xyz" the first thing you do is look for seeders in the swarm for that file. If non exists, then you initiate an http download from docker to get the image. Now you've got fast distribution with low external network traffic. Not sure how this would play with Cloud provider pricing, though. I don't believe AWS would be too happy seeing their services turned into BT swarms :)
- yjftsjthsd-h 6y ago
  > How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy. 1. Only some companies work like that. 2. I'd expect it to work like a webtorrent; try to download by p2p, but if that fails then fall back to HTTP.
closeparen 6y ago
https://github.com/uber/kraken https://github.com/uber/kraken
jacques_chester 6y ago
> Honest question, but why can't docker use something like bittorrent to download images? Docker will be limiting manifest operations, not actual blob transmissions.

saagarjha 6y ago

> Most of us download our OS via torrents only I mean, I only do it to stick it to the people who claim torrents can only be used for piracy; I think most people prefer the simplicity of direct downloads though…

bigmanwalter 6y ago
I can usually max out my connection speed with torrents. This is rarely the case with direct downloads.
- saagarjha 6y ago
  Guess I just have slow internet :(
- stingraycharles 6y ago
  The start up time of a torrent is typically so long though, by the time it has connected to all peers and is downloading I have already downloaded the iso with a direct download anyway.
  - stonewareslord 6y ago
    There's a feature of linx-server that provides uploaded files with a torrent URL as well as a regular download URL. I believe the torrent client tries fetching data from peers as well as from linx (http). https://github.com/andreimarcu/linx-server https://github.com/andreimarcu/linx-server
  - namibj 6y ago
    Soon, when QUIC is widely supported by the swarm (just needs updating to the current version), IPFS should be better than torrents for start-up delay, and much more importantly, it allows for sharing data across "torrents" when using content-defined chunking via rabin or buzhash. This means that things like common larger binaries get shared between images that include them, which should greatly increase the average amount of seeders for the chunks that make up an image.
- skunkworker 6y ago
  On symmetric gigabit fiber only a few services, mainly steam and battle.net have ever given me 90mbyte/s download speeds. A surprising amount will limit to 500mbit or less, even if you have the download pipe for it.
Crazyontap 6y ago
At least for Ubuntu and popular distros like mint you get super high speeds on torrent maybe because you have a peer in the same region. Afaik Windows is also using this to install updates where it shares the download with others in the region (1) using p2p (though i may be wrong since i don't use Windows anymore) (1) https://www.itproportal.com/amp/news/how-to-stop-windows-10-sharing-updates-with-your-internet-connection/ https://www.itproportal.com/amp/news/how-to-stop-windows-10-...
Nullabillity 6y ago
Docker images already need special handling since you download the layers separately and reassemble them. Going from that to full BitTorrent should be transparent to the users. In fact, there already exist several implementations of it for Docker![0, 1, 2] [0]: https://coreos.com/blog/torrent-pulls https://coreos.com/blog/torrent-pulls [1]: https://d7y.io/en-us/ https://d7y.io/en-us/ [2]: https://github.com/uber/kraken https://github.com/uber/kraken

bsamuels 6y ago

The hardest part of that would be verifying image authenticity. Google Cloud uses an adjacent feature called binary authorization. When turned on, only images that are signed by a given authority (usually your ci/cd instruments) can be run inside your Kubernetes cluster. Binary authorization may be a good starting point for someone trying to make bittorrent distributed images a usable thing.

sneak 6y ago
> The hardest part of that would be verifying image authenticity. That's exactly what Bittorrent does with its hash tree. You'd get the root hash (extremely tiny) from Docker Hub, and the rest of the metadata, as well as the data blocks, from the swarm. The authenticity is all handled by the TLS that serves you the root infohash from Docker Hub. It's a Merkle tree: the root hash is for the metadata, which is a list of hashes of the blocks.
- im3w1l 6y ago
  If this reads like greek, let me dumb it down. You get the hash of the final result from the trusted server and hash is checked. Because of this you will never get an invalid image. There are also some clever tricks to make sure no one can force you to start over from scratch by sending wrong data. But that's more of a detail.
pixl97 6y ago
Why, the website could still have the hash for the image which is only a few KB versus hundreds of megabytes or even gigabytes. Just have the bit-docker app check the hash before executing.
jtsiskin 6y ago
What is the issue here? Is the torrent checksum (provided by docker) not enough?
- bsamuels 6y ago
  Let's say someone hacks a maintainer for the Ubuntu base image. The hacker publishes a new version of the base image with a backdoor. When the backdoor is detected, you now need a revocation system so the distribution of the malicious image will die. You can theoretically do this on the tracker level, but people may build other trackers that may not propagate the changes.

user5994461 6y ago

How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy.

Crazyontap 6y ago
- Its an alternative download solution - There is no rate limiting for paid accounts or companies from what I see in the article.
znpy 6y ago
The company proxy would have to be modified to allow torrent traffic, I guess?
cogman10 6y ago
You could have a mixed mode pretty easily. Heck, it would even be pretty efficient. For your own company, you'd host a swarm that is firewalled in. Then when someone says "I want image xyz" the first thing you do is look for seeders in the swarm for that file. If non exists, then you initiate an http download from docker to get the image. Now you've got fast distribution with low external network traffic. Not sure how this would play with Cloud provider pricing, though. I don't believe AWS would be too happy seeing their services turned into BT swarms :)
yjftsjthsd-h 6y ago
> How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy. 1. Only some companies work like that. 2. I'd expect it to work like a webtorrent; try to download by p2p, but if that fails then fall back to HTTP.

closeparen 6y ago

https://github.com/uber/kraken https://github.com/uber/kraken

jacques_chester 6y ago

> Honest question, but why can't docker use something like bittorrent to download images? Docker will be limiting manifest operations, not actual blob transmissions.