5 ms·

For image and layer manipulation, crane is awesome - as is the underlying go-containerregistry library. It lets you add new layers, or edit any metadata (env v

For image and layer manipulation, crane is awesome - as is the underlying go-containerregistry library. It lets you add new layers, or edit any metadata (env vars, labels, entrypoint, etc) in existing images. You can also "flatten" an image with multiple layers into a single layer. Additionally you can "rebase" an image (re-apply your changes onto a new/updated base image). It does all this directly in the registry, so no docker needed (though it's still useful for creating the original image). https://github.com/google/go-containerregistry/blob/main/cmd/crane/recipes.md https://github.com/google/go-containerregistry/blob/main/cmd... (updated: better link)

pbowyer 3y ago
Is there any performance benefit to having fewer layers? My understanding is that there's no gain by merging layers as the size of the image remains constant.
- natebc 3y ago
  some startup performance savings in fewer http requests to fetch the image. small for sure but it's something?
  - electroly 3y ago
    In practice I've found the performance savings often goes the other way--for large (multi-GB) images it's faster to split it up into more layers that it can download in parallel from the registry. It won't parallelize the download of a single layer and in EC2+ECR you won't get particularly good throughput with a single layer.
  - whirlwin 3y ago
    Depends. If you would have to fetch a big layer often because of updates, that's not good. But if what is changing frequently is in a smaller layer, it will be more favorable
- mcpherrinm 3y ago
  If files are overwritten or removed in a lower layer, there can be size savings from that.
- apt-get 3y ago
  There are some useful cases — for example, if you're taking a rather bloated image as a base and trimming it down with `rm` commands, those will be saved as differential layers, which will not reduce the size of the final image in the slightest. Only merging will actually "register" these deletions.
- miquong 3y ago
  Eventually, once zstd support gets fully supported, and tiny gzip compression windows are not a limitation, then compressing a full layer would almost certainly have a better ratio over several smaller layers https://github.com/opencontainers/image-spec/issues/803 https://github.com/opencontainers/image-spec/issues/803
  - pbowyer 3y ago
    Is it coming? That ticket doesn't fill me with hope, given its age and the disagreements over backwards compatibility.
- fishpen0 3y ago
  Less performance and more security. Lots of ameteur images use a secret file or inadvertently store a secret to a layer without realizing an rm or other process in another layer doesn't actually eliminate it. If the final step of your build squashes the filesystem flat again you can remove a lot of potentially exposed metadata and secrets stored in intermediate layers
- momothereal 3y ago
  I'm working on a tool that does the opposite: to split layers into smaller, deterministic deltas.
- yrro 3y ago
  If you've got a 50 layer image then each time you open a file, I believe the kernel has to look for that file in all 50 layers before it can fail with ENOENT.
  - naikrovek 3y ago
    That seems ripe for optimization, if true, especially since those layers are all immutable.
    - yrro 3y ago
      A container runtime could optimize for speed by unpacking all those layers one by one into a single lower directory for the container to use; but at the cost of using lots of disk space, since those layers would no longer be shared between different containers.
  - cmckn 3y ago
    It depends on your OCI engine; but this isn’t the case with containers. Each layer is successively “unpacked” upon a “snapshot”, from which containers are created.
    - yrro 3y ago
      Interesting. Certainly with podman I observe that the layers are combined by overlayfs. I build this Dockerfile into a container with 'podman build -t test .': FROM registry.access.redhat.com/ubi9/ubi-minimal RUN touch /file1 RUN touch /file2 RUN touch /file3 Then run it with 'podman run --rm -it test': Then I can inspect the container's storage configuration: $ podman inspect -l | jq '.[].GraphDriver' { "Name": "overlay", "Data": { "LowerDir": "/home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff:/home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff:/home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff:/home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff", "MergedDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/merged", "UpperDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/diff", "WorkDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/work" } } Note that there are four directories in the LowerDir, and examining their contents reveals that there's one per layer in my 'test' image: $ podman inspect -l | jq '.[].GraphDriver.Data.LowerDir' -r | tr : '\n' | xargs -rt -n1 ls ls /home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff file3 run/ ls /home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff file2 run/ ls /home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff file1 run/ ls /home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff afs/ bin@ boot/ dev/ etc/ home/ lib@ lib64@ lost+found/ media/ mnt/ opt/ proc/ root/ run/ sbin/ srv/ sys/ tmp/ usr/ var/ Same for a system (non-rootless) container: # podman inspect systemd-oxidized | jq '.[].GraphDriver.Data.LowerDir' -r | tr ':' '\n' | xargs -rt -n 1 ls -F ls -F /var/lib/containers/storage/overlay/062693a4d8fe24055645ee1c11bbc542507ba3fe101ed4e5b76b6457752d3ac5/diff home/ ls -F /var/lib/containers/storage/overlay/9d9c6a906aff40160f924c4fec96692a6fc59f27316b8df5ee2616091008a2fe/diff home/ ls -F /var/lib/containers/storage/overlay/e35e4cf6ce1057979baee991e609f82afb85360560dc8bf36ab069d8e8288f5f/diff home/ ls -F /var/lib/containers/storage/overlay/da5189fd0e740204d7a30e4b59b012251e8d36a1db488bf47ede61247a948fff/diff home/ ls -F /var/lib/containers/storage/overlay/702d48c0e8afce7cf5ce049939d8fdb596755e709a3735bbbf4900ec65c45529/diff etc/ home/ var/ ls -F /var/lib/containers/storage/overlay/877031909d6cf6038e0e0295c34f14932b71903835b3eb13ae6bf23396d65daf/diff etc/ usr/ var/ ls -F /var/lib/containers/storage/overlay/b90fc2ebe1d7c19e5ee43033b8e8be95210b4e3a0012bb54db327f6d52e89d12/diff tmp/ ls -F /var/lib/containers/storage/overlay/aa2e131948d26d04f48f7c01c57aa98e2a308e4fc0d707c9532d30fa4ab89f5d/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/6af6a203c83f697aa35b5007b5678b8c2b052ab482d9d7f7e5b322cd74682441/diff root/ tmp/ usr/ var/ ls -F /var/lib/containers/storage/overlay/b237f20a22db5d282838c5a7c9d78f7d6e648d9b223a328fa8b8078e9c6b5bbe/diff ls -F /var/lib/containers/storage/overlay/7d1bc5d46561795a4f7058e4098faa345e119617292761933e2ad7f185dbd45b/diff ls -F /var/lib/containers/storage/overlay/88d52756a706920fbcb38cbbb6eaf43ea09be0045361ffe6920185ca7179a1d9/diff tmp/ ls -F /var/lib/containers/storage/overlay/0d0dedc3e4f8d7f39b9a0c78d1cf380ad1161023a49bd74ab8b51626f6d21f19/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/d74968407a57d753c2ae232ea26e0e2d7fed30683a67a993b814b71684817eaa/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/885b215dc2a8304a58a6d35a5668c74d3d80f27dacf56b97a3b8b13c1a423b33/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/a4edd3819e3655d0ca90ac898a4a2b91ec50c83cbf61862186e10ae1b44cbcc0/diff etc/ usr/ var/ ls -F /var/lib/containers/storage/overlay/954dcca00292635597e4c5ee9ebc35bf5fcc15a50dfd8f176df7b4c15e695e10/diff bd_build/ etc/ root/ run/ usr/ var/ ls -F /var/lib/containers/storage/overlay/fb670aa62e0f4f71382ef61a742e8ab1ad5d345b95267b0adc18c10e6c87f13b/diff bd_build/ ls -F /var/lib/containers/storage/overlay/7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274/diff bin@ boot/ dev/ etc/ home/ lib@ lib32@ lib64@ libx32@ media/ mnt/ opt/ proc/ root/ run/ sbin@ srv/ sys/ tmp/ usr/ var/ ('podman system info' confirms that .store.graphDriverName == "overlay") I'd be interested to see what this looks like with docker?
- dayjaby 3y ago
  One simple case where the resulting image is bigger than necessary: ``` COPY ./package.deb /tmp/package.deb RUN dpkg -i /tmp/*.deb && rm -rf /tmp/*.deb ``` This results in two layers, with one layer containing a huge file, thus being part of the final image if you don't do multi-stage builds.
lyxell 3y ago
This is a great recommendation. It is worth noting that unlike Docker, crane is root- and daemonless which makes it work great in Nix (it's called 'crane' in the Nix repository). This allows for Nix to be used to manage dependencies for both building (e.g. Go) as well as packaging and deploying (e.g. gnu tar, crane).

pbowyer 3y ago

Is there any performance benefit to having fewer layers? My understanding is that there's no gain by merging layers as the size of the image remains constant.

natebc 3y ago
some startup performance savings in fewer http requests to fetch the image. small for sure but it's something?
- electroly 3y ago
  In practice I've found the performance savings often goes the other way--for large (multi-GB) images it's faster to split it up into more layers that it can download in parallel from the registry. It won't parallelize the download of a single layer and in EC2+ECR you won't get particularly good throughput with a single layer.
- whirlwin 3y ago
  Depends. If you would have to fetch a big layer often because of updates, that's not good. But if what is changing frequently is in a smaller layer, it will be more favorable
mcpherrinm 3y ago
If files are overwritten or removed in a lower layer, there can be size savings from that.
apt-get 3y ago
There are some useful cases — for example, if you're taking a rather bloated image as a base and trimming it down with `rm` commands, those will be saved as differential layers, which will not reduce the size of the final image in the slightest. Only merging will actually "register" these deletions.
miquong 3y ago
Eventually, once zstd support gets fully supported, and tiny gzip compression windows are not a limitation, then compressing a full layer would almost certainly have a better ratio over several smaller layers https://github.com/opencontainers/image-spec/issues/803 https://github.com/opencontainers/image-spec/issues/803
- pbowyer 3y ago
  Is it coming? That ticket doesn't fill me with hope, given its age and the disagreements over backwards compatibility.
fishpen0 3y ago
Less performance and more security. Lots of ameteur images use a secret file or inadvertently store a secret to a layer without realizing an rm or other process in another layer doesn't actually eliminate it. If the final step of your build squashes the filesystem flat again you can remove a lot of potentially exposed metadata and secrets stored in intermediate layers
momothereal 3y ago
I'm working on a tool that does the opposite: to split layers into smaller, deterministic deltas.
yrro 3y ago
If you've got a 50 layer image then each time you open a file, I believe the kernel has to look for that file in all 50 layers before it can fail with ENOENT.
- naikrovek 3y ago
  That seems ripe for optimization, if true, especially since those layers are all immutable.
  - yrro 3y ago
    A container runtime could optimize for speed by unpacking all those layers one by one into a single lower directory for the container to use; but at the cost of using lots of disk space, since those layers would no longer be shared between different containers.
- cmckn 3y ago
  It depends on your OCI engine; but this isn’t the case with containers. Each layer is successively “unpacked” upon a “snapshot”, from which containers are created.
  - yrro 3y ago
    Interesting. Certainly with podman I observe that the layers are combined by overlayfs. I build this Dockerfile into a container with 'podman build -t test .': FROM registry.access.redhat.com/ubi9/ubi-minimal RUN touch /file1 RUN touch /file2 RUN touch /file3 Then run it with 'podman run --rm -it test': Then I can inspect the container's storage configuration: $ podman inspect -l | jq '.[].GraphDriver' { "Name": "overlay", "Data": { "LowerDir": "/home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff:/home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff:/home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff:/home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff", "MergedDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/merged", "UpperDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/diff", "WorkDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/work" } } Note that there are four directories in the LowerDir, and examining their contents reveals that there's one per layer in my 'test' image: $ podman inspect -l | jq '.[].GraphDriver.Data.LowerDir' -r | tr : '\n' | xargs -rt -n1 ls ls /home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff file3 run/ ls /home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff file2 run/ ls /home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff file1 run/ ls /home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff afs/ bin@ boot/ dev/ etc/ home/ lib@ lib64@ lost+found/ media/ mnt/ opt/ proc/ root/ run/ sbin/ srv/ sys/ tmp/ usr/ var/ Same for a system (non-rootless) container: # podman inspect systemd-oxidized | jq '.[].GraphDriver.Data.LowerDir' -r | tr ':' '\n' | xargs -rt -n 1 ls -F ls -F /var/lib/containers/storage/overlay/062693a4d8fe24055645ee1c11bbc542507ba3fe101ed4e5b76b6457752d3ac5/diff home/ ls -F /var/lib/containers/storage/overlay/9d9c6a906aff40160f924c4fec96692a6fc59f27316b8df5ee2616091008a2fe/diff home/ ls -F /var/lib/containers/storage/overlay/e35e4cf6ce1057979baee991e609f82afb85360560dc8bf36ab069d8e8288f5f/diff home/ ls -F /var/lib/containers/storage/overlay/da5189fd0e740204d7a30e4b59b012251e8d36a1db488bf47ede61247a948fff/diff home/ ls -F /var/lib/containers/storage/overlay/702d48c0e8afce7cf5ce049939d8fdb596755e709a3735bbbf4900ec65c45529/diff etc/ home/ var/ ls -F /var/lib/containers/storage/overlay/877031909d6cf6038e0e0295c34f14932b71903835b3eb13ae6bf23396d65daf/diff etc/ usr/ var/ ls -F /var/lib/containers/storage/overlay/b90fc2ebe1d7c19e5ee43033b8e8be95210b4e3a0012bb54db327f6d52e89d12/diff tmp/ ls -F /var/lib/containers/storage/overlay/aa2e131948d26d04f48f7c01c57aa98e2a308e4fc0d707c9532d30fa4ab89f5d/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/6af6a203c83f697aa35b5007b5678b8c2b052ab482d9d7f7e5b322cd74682441/diff root/ tmp/ usr/ var/ ls -F /var/lib/containers/storage/overlay/b237f20a22db5d282838c5a7c9d78f7d6e648d9b223a328fa8b8078e9c6b5bbe/diff ls -F /var/lib/containers/storage/overlay/7d1bc5d46561795a4f7058e4098faa345e119617292761933e2ad7f185dbd45b/diff ls -F /var/lib/containers/storage/overlay/88d52756a706920fbcb38cbbb6eaf43ea09be0045361ffe6920185ca7179a1d9/diff tmp/ ls -F /var/lib/containers/storage/overlay/0d0dedc3e4f8d7f39b9a0c78d1cf380ad1161023a49bd74ab8b51626f6d21f19/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/d74968407a57d753c2ae232ea26e0e2d7fed30683a67a993b814b71684817eaa/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/885b215dc2a8304a58a6d35a5668c74d3d80f27dacf56b97a3b8b13c1a423b33/diff root/ usr/ var/ ls -F /var/lib/containers/storage/overlay/a4edd3819e3655d0ca90ac898a4a2b91ec50c83cbf61862186e10ae1b44cbcc0/diff etc/ usr/ var/ ls -F /var/lib/containers/storage/overlay/954dcca00292635597e4c5ee9ebc35bf5fcc15a50dfd8f176df7b4c15e695e10/diff bd_build/ etc/ root/ run/ usr/ var/ ls -F /var/lib/containers/storage/overlay/fb670aa62e0f4f71382ef61a742e8ab1ad5d345b95267b0adc18c10e6c87f13b/diff bd_build/ ls -F /var/lib/containers/storage/overlay/7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274/diff bin@ boot/ dev/ etc/ home/ lib@ lib32@ lib64@ libx32@ media/ mnt/ opt/ proc/ root/ run/ sbin@ srv/ sys/ tmp/ usr/ var/ ('podman system info' confirms that .store.graphDriverName == "overlay") I'd be interested to see what this looks like with docker?
dayjaby 3y ago
One simple case where the resulting image is bigger than necessary: ``` COPY ./package.deb /tmp/package.deb RUN dpkg -i /tmp/*.deb && rm -rf /tmp/*.deb ``` This results in two layers, with one layer containing a huge file, thus being part of the final image if you don't do multi-stage builds.

lyxell 3y ago

This is a great recommendation. It is worth noting that unlike Docker, crane is root- and daemonless which makes it work great in Nix (it's called 'crane' in the Nix repository). This allows for Nix to be used to manage dependencies for both building (e.g. Go) as well as packaging and deploying (e.g. gnu tar, crane).