3 ms·

> A change to a lower layer invalidates all layers above it Does it have to? It seems it should be possible to diff the layers and only invalidate if there are

> A change to a lower layer invalidates all layers above it Does it have to? It seems it should be possible to diff the layers and only invalidate if there are conflicts.

__MatrixMan__ 2y ago
The way Dockerfiles work, yes I think it does need to do this. It wouldn't be a matter of "conflicts" but rather of assembling containers with the wrong data. Imagine a Dockerfile like so: RUN echo 1 > A RUN echo "$(cat A) + 1" | bc > B So that's two layers each with one file. If, in a later version, the first command changes to `echo 3 > A` then the contents of B should become "4", even though the second command didn't change. That is, neither layer can be reused because the layers depend on each other. But maybe there's no dependency. If your Dockerfile is like this: RUN echo 1 > A RUN echo 2 > B Then the second layer could in theory be re-used when the first layer changes, and not built/pushed/downloaded a second time. RUN echo 3 > A # new RUN echo 2 > B # no dependency on layer 1, can be reused But docker doesn't do this. It plays it safe and unnecessarily rebuilds both layers anyway. And since these files end up with timestamps, the hashes of the layers differ, so both layers are consequently reuploaded and redownloaded. Build tools like nix and bazel require more of the user. You can't just run commands all willy nilly, you have to tell them more info about which things depend on which other things. But the consequence is that instead of a list of layers you end up with a richer representation of how dependency works in your project (I guess it's a DAG). Armed with this, when you try to build the next version of something, you only have to rebuild the parts that actually depend on the changes. Whether the juice is worth the squeeze is an open question. I think it is.

> A change to a lower layer invalidates all layers above it Does it have to? It seems it should be possible to diff the layers and only invalidate if there are conflicts.

__MatrixMan__ 2y ago

The way Dockerfiles work, yes I think it does need to do this. It wouldn't be a matter of "conflicts" but rather of assembling containers with the wrong data. Imagine a Dockerfile like so: RUN echo 1 > A RUN echo "$(cat A) + 1" | bc > B So that's two layers each with one file. If, in a later version, the first command changes to `echo 3 > A` then the contents of B should become "4", even though the second command didn't change. That is, neither layer can be reused because the layers depend on each other. But maybe there's no dependency. If your Dockerfile is like this: RUN echo 1 > A RUN echo 2 > B Then the second layer could in theory be re-used when the first layer changes, and not built/pushed/downloaded a second time. RUN echo 3 > A # new RUN echo 2 > B # no dependency on layer 1, can be reused But docker doesn't do this. It plays it safe and unnecessarily rebuilds both layers anyway. And since these files end up with timestamps, the hashes of the layers differ, so both layers are consequently reuploaded and redownloaded. Build tools like nix and bazel require more of the user. You can't just run commands all willy nilly, you have to tell them more info about which things depend on which other things. But the consequence is that instead of a list of layers you end up with a richer representation of how dependency works in your project (I guess it's a DAG). Armed with this, when you try to build the next version of something, you only have to rebuild the parts that actually depend on the changes. Whether the juice is worth the squeeze is an open question. I think it is.