4 ms·

layers are not delta transfers in my opinion, they are just new copies or removed files. its like saying sftp is delta transfer because you don't have to send y

by bepvte 8y ago

layers are not delta transfers in my opinion, they are just new copies or removed files. its like saying sftp is delta transfer because you don't have to send your entire disk image every time. I love layers and dockers very nice reusing of base layers and such but i feel like delta transfers would dramatically improve my workflow and upload times.

lstamour 8y ago
My understanding is the on-disk storage might not be delta, as tar files, but the transfer on the wire is... could be wrong, but I think they intentionally made delta-transfer a docker registry network concern? I suppose if you wanted delta storage on-disk that again could be an implementation concern, knowing where to pull and reassemble the bits into a standard image? Oh and it uses gzip compression on the wire and possibly elsewhere, again a form of delta compression.
Confiks 8y ago
> its like saying sftp is delta transfer because you don't have to send your entire disk image every time I see your point, but the comparison to sftp seems really off. Overlay2 [1] indeed works on whole file differences, just as sftp would do. Additionally, I don't think making actual byte-level deltas would yield you much improvement on container size. It would also increase access time unless you keep a cache that doubles storage requirements per image. Dockerfiles primarily add files rather than change them, and actual changed files are often small. [1] https://docs.docker.com/storage/storagedriver/overlayfs-driver/ https://docs.docker.com/storage/storagedriver/overlayfs-driv...
infogulch 8y ago
> i feel like delta transfers would dramatically improve my workflow and upload times If you think delta transfers would help, you're saying you have a bunch of data that doesn't change between image builds, but you have to re-upload all that data every time. This actually sounds like a perfect use-case for image layers. If you can rearrange your Dockerfile to put the steps where files change more rarely before steps that change frequently, docker will automatically cache the earlier steps and not reupload if that layer already exists. Image layers are at their heart already an implementation of a cache + delta transfer system. (Compared to some other delta systems it trades storage efficiency for computational efficiency.) You can get small delta transfers, but you have to work with docker via your Dockerfile to make it happen. For example, that could mean: moving steps up or down the Dockerfile so more frequently changed layers come later; splitting a step into two steps to exploit the previous; making a step deterministic so you don't have to force rebuild; use multi-stage builds to make all of these easier to implement.

by bepvte 8y ago

lstamour 8y ago

My understanding is the on-disk storage might not be delta, as tar files, but the transfer on the wire is... could be wrong, but I think they intentionally made delta-transfer a docker registry network concern? I suppose if you wanted delta storage on-disk that again could be an implementation concern, knowing where to pull and reassemble the bits into a standard image? Oh and it uses gzip compression on the wire and possibly elsewhere, again a form of delta compression.

Confiks 8y ago

> its like saying sftp is delta transfer because you don't have to send your entire disk image every time I see your point, but the comparison to sftp seems really off. Overlay2 [1] indeed works on whole file differences, just as sftp would do. Additionally, I don't think making actual byte-level deltas would yield you much improvement on container size. It would also increase access time unless you keep a cache that doubles storage requirements per image. Dockerfiles primarily add files rather than change them, and actual changed files are often small. [1] https://docs.docker.com/storage/storagedriver/overlayfs-driver/ https://docs.docker.com/storage/storagedriver/overlayfs-driv...

infogulch 8y ago

> i feel like delta transfers would dramatically improve my workflow and upload times If you think delta transfers would help, you're saying you have a bunch of data that doesn't change between image builds, but you have to re-upload all that data every time. This actually sounds like a perfect use-case for image layers. If you can rearrange your Dockerfile to put the steps where files change more rarely before steps that change frequently, docker will automatically cache the earlier steps and not reupload if that layer already exists. Image layers are at their heart already an implementation of a cache + delta transfer system. (Compared to some other delta systems it trades storage efficiency for computational efficiency.) You can get small delta transfers, but you have to work with docker via your Dockerfile to make it happen. For example, that could mean: moving steps up or down the Dockerfile so more frequently changed layers come later; splitting a step into two steps to exploit the previous; making a step deterministic so you don't have to force rebuild; use multi-stage builds to make all of these easier to implement.