3 ms·

It seems like a good time to point out that oci images' layer-based caching system is incredibly bandwidth inefficient. A change to a lower layer invalidates a

It seems like a good time to point out that oci images' layer-based caching system is incredibly bandwidth inefficient. A change to a lower layer invalidates all layers above it, regardless of whether there's actually any dependency on the changed data. With a competent caching strategy (the sort of thing you'd set up with nix or bazel) it's often faster to send the git SHA and build the image on the other end than it is to move built images around. This is because 99% of that image you're downloading or pushing is probably already on the target machine, but the images don't contain enough metadata to tell you where that 1% is. A build tool, by contrast, understands inputs and outputs. If the inputs haven't changed, it can just use the outputs which are still lying around from last time.

silverwind 2y ago
> A change to a lower layer invalidates all layers above it Does it have to? It seems it should be possible to diff the layers and only invalidate if there are conflicts.
- __MatrixMan__ 2y ago
  The way Dockerfiles work, yes I think it does need to do this. It wouldn't be a matter of "conflicts" but rather of assembling containers with the wrong data. Imagine a Dockerfile like so: RUN echo 1 > A RUN echo "$(cat A) + 1" | bc > B So that's two layers each with one file. If, in a later version, the first command changes to `echo 3 > A` then the contents of B should become "4", even though the second command didn't change. That is, neither layer can be reused because the layers depend on each other. But maybe there's no dependency. If your Dockerfile is like this: RUN echo 1 > A RUN echo 2 > B Then the second layer could in theory be re-used when the first layer changes, and not built/pushed/downloaded a second time. RUN echo 3 > A # new RUN echo 2 > B # no dependency on layer 1, can be reused But docker doesn't do this. It plays it safe and unnecessarily rebuilds both layers anyway. And since these files end up with timestamps, the hashes of the layers differ, so both layers are consequently reuploaded and redownloaded. Build tools like nix and bazel require more of the user. You can't just run commands all willy nilly, you have to tell them more info about which things depend on which other things. But the consequence is that instead of a list of layers you end up with a richer representation of how dependency works in your project (I guess it's a DAG). Armed with this, when you try to build the next version of something, you only have to rebuild the parts that actually depend on the changes. Whether the juice is worth the squeeze is an open question. I think it is.

silverwind 2y ago

> A change to a lower layer invalidates all layers above it Does it have to? It seems it should be possible to diff the layers and only invalidate if there are conflicts.

__MatrixMan__ 2y ago
The way Dockerfiles work, yes I think it does need to do this. It wouldn't be a matter of "conflicts" but rather of assembling containers with the wrong data. Imagine a Dockerfile like so: RUN echo 1 > A RUN echo "$(cat A) + 1" | bc > B So that's two layers each with one file. If, in a later version, the first command changes to `echo 3 > A` then the contents of B should become "4", even though the second command didn't change. That is, neither layer can be reused because the layers depend on each other. But maybe there's no dependency. If your Dockerfile is like this: RUN echo 1 > A RUN echo 2 > B Then the second layer could in theory be re-used when the first layer changes, and not built/pushed/downloaded a second time. RUN echo 3 > A # new RUN echo 2 > B # no dependency on layer 1, can be reused But docker doesn't do this. It plays it safe and unnecessarily rebuilds both layers anyway. And since these files end up with timestamps, the hashes of the layers differ, so both layers are consequently reuploaded and redownloaded. Build tools like nix and bazel require more of the user. You can't just run commands all willy nilly, you have to tell them more info about which things depend on which other things. But the consequence is that instead of a list of layers you end up with a richer representation of how dependency works in your project (I guess it's a DAG). Armed with this, when you try to build the next version of something, you only have to rebuild the parts that actually depend on the changes. Whether the juice is worth the squeeze is an open question. I think it is.