4 ms·

This is what Singularity (and to a lesser extent, LXD) do. The main problem with this is that you don't get de-duplication of transfer (or really of storage) --

by cyphar 8y ago

This is what Singularity (and to a lesser extent, LXD) do. The main problem with this is that you don't get de-duplication of transfer (or really of storage) -- any small change in your published rootfs and you'd have to re-download the whole thing. In addition, it requires that the system you're mounting it on supports the filesystem you use (and that the admins are happy using that filesystem). There is also a potential security risk -- since filesystem drivers are generally not secured against potentially malicious sources (there are plenty of attacks that have been found against the big Linux filesystems when you attack them with un-trusted filesystem data). This is one of the reasons auto-mounting USBs is generally seen as a bad security practice. Don't get me wrong, there is a _huge_ benefit to using your runtime format as your image distribution format. But there are downsides that are non-trivial to work around. I am thinking about how to bridge that gap though.

ofrzeta 8y ago
What about some kind of binary delta diffs such as bsdiff?
- cyphar 8y ago
  Yes, and this is what LXD does. I think I mentioned it in the article, but basically the issue is that it requires one of: 1. A clever server, which asks you which version do you have so it can generate a diff for you. This has quite a few drawbacks (storage and processing costs as well as making it harder to verify that the image you end up with is what was signed by the original developers). But this will guarantee that you will always get de-duplication. 2. Or you could pre-generate diffs for a specific set of versions, which means it's a lottery whether or not users actually get transfer de-duplication. If you generate a diff for _every_ version you're now back to storage costs (and processing costs on the developer side that increase with each version of the container image you have). You could make it so that the diffs only step you forward one version rather than instantly get you the latest, but then you now have clients having to pull many binary diffs again. This system has existed for a long time with BSD as well as distributions having delta-RPMs (or the equivalent for debs). It works _okay_ but it's far from ideal, and the other negatives of using loopback filesystems only make it less ideal. In my view, dumb formats are best.
  - bruce_one 8y ago
    Using [zsync](http://zsync.moria.org.uk/ http://zsync.moria.org.uk/) is an alternative option from my understanding? I could be technically inaccurate, but my understanding is that it's rsync but with the server serving a metadata file which allows the rsync-diffing to happen from the client side rather than the server side - hence no clever server required. It also doesn't require diffing particular revisions; but only the different blocks will be fetched. It does require server the metadata file, but they're note very large afaik.

by cyphar 8y ago

ofrzeta 8y ago

What about some kind of binary delta diffs such as bsdiff?

cyphar 8y ago
Yes, and this is what LXD does. I think I mentioned it in the article, but basically the issue is that it requires one of: 1. A clever server, which asks you which version do you have so it can generate a diff for you. This has quite a few drawbacks (storage and processing costs as well as making it harder to verify that the image you end up with is what was signed by the original developers). But this will guarantee that you will always get de-duplication. 2. Or you could pre-generate diffs for a specific set of versions, which means it's a lottery whether or not users actually get transfer de-duplication. If you generate a diff for _every_ version you're now back to storage costs (and processing costs on the developer side that increase with each version of the container image you have). You could make it so that the diffs only step you forward one version rather than instantly get you the latest, but then you now have clients having to pull many binary diffs again. This system has existed for a long time with BSD as well as distributions having delta-RPMs (or the equivalent for debs). It works _okay_ but it's far from ideal, and the other negatives of using loopback filesystems only make it less ideal. In my view, dumb formats are best.
- bruce_one 8y ago
  Using [zsync](http://zsync.moria.org.uk/ http://zsync.moria.org.uk/) is an alternative option from my understanding? I could be technically inaccurate, but my understanding is that it's rsync but with the server serving a metadata file which allows the rsync-diffing to happen from the client side rather than the server side - hence no clever server required. It also doesn't require diffing particular revisions; but only the different blocks will be fetched. It does require server the metadata file, but they're note very large afaik.