7 ms·

> Everything after that removes the manifest files and any temporary files downloaded during this command. It's necessary to remove all these files in this comm

by jchw 4y ago

> Everything after that removes the manifest files and any temporary files downloaded during this command. It's necessary to remove all these files in this command to keep the size of the Docker image to a minimum. Smaller Dockerfiles mean faster deployments. It isn't explicitly explained, but the reason why it must be in this command and not separated out is because each command in a dockerfile creates a new "layer". Removing the files in another command will work, but it does nothing to decrease the overall image size: as far as whatever filesystem driver you're using is concerned, deleting files from earlier layers is just masking them, whereas deleting them before creating the layer prevents them from ever actually being stored.

viraptor 4y ago
If you want to check your images for some common leftover files in all the layers, I made an app for that: https://github.com/viraptor/cruftspy https://github.com/viraptor/cruftspy
cvccvroomvroom 4y ago
Every command in a dockerfile creates a layer, and most Dockerfile builders cache each if that line hasn't changed. Dynamic callouts to run updates or check things on the web won't rerun.
mattficke 4y ago
The older “Docker without Docker” blogpost linked from there goes into that, it’s one of the best deep dives into containers (and honestly one of the best pieces of technical writing full stop) I’ve come across. https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
emerongi 4y ago
If you only care about the final image size, then `docker build --squash` squashes the layers for you as well.
- jchw 4y ago
  Definitely, although it's worth noting that while the image size will be smaller, it will get rid of the benefits of sharing base layers. Having less redundant layers lets you save most of the space without losing any of the benefits of sharing slices. I think that is the main reason why this is not usually done.
- gkfasdfasdf 4y ago
  --squash is still experimental, I believe multi-stage images are the new best practice here.
  - 411111111111111 4y ago
    in case anyone doesn't know what that means, its basically this kind of dockerfile FROM the_source_image as builder RUN build.sh FROM the_source_image COPY --from=builder /app/artifacts /app/ CMD .... i'm not sure if you can really call it the new best practice though, its been the default for ... a very long time at this point.
    - jchw 4y ago
      Typically I wind up using a different source image for the builder that ideally has (most of) the toolchain bits needed, but the same runtime base as the final image. (For Go, go:alpine and alpine work well. I'm aware alpine/musl is not technically supported by Go, but I have yet to hit issues in prod with it, so I guess I'll keep taking that gamble.)
      - deleted 4y ago
        [deleted]
    - KronisLV 4y ago
      I take advantage of multi-stage builds, however I still think that the layer system could have some nice improvements done to it. For example, say I have my own Ubuntu image that is based on one of the official ones, but adds a bit of common configuration or tools and so on, on which I then build my own Java image using the package manager (not unlike what Bitnami do with their minideb, on which they then base their PostgreSQL and most other container images). So I might have something like the following in the Ubuntu image Dockerfile: RUN apt-get update && apt-get install -y \ curl wget \ net-tools inetutils-ping dnsutils \ supervisor \ && apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/* But then, if I want to install additional software, I need to fetch the package list anew downstream: FROM my-own-repo/ubuntu RUN apt-get update && apt-get install -y \ openjdk-17-jdk-headless \ && apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/* As opposed to being able to just leave the cache files in the previous layers/images, then remove them in a later layer and just do something like: docker build -t my_optimized_java_image -f java.Dockerfile --purge-deleted-files . or maybe docker build -t my_regular_java_image -f java.Dockerfile . purge-deleted-files -t my_regular_java_image -o my_optimized_java_image Which would then work backwards from the last layer and create copies of all of the layers where files have been removed/masked (in the later layers) to use instead of the originals. Thus if I'd have 10 different images that need to use apt to install stuff while building them, I could leave the cache in my own Ubuntu image and then just remove it for whatever I want to consider the "final" images that I'll ship, which would then alter the contents of the included layers to purge deleted files. There's little reason why these optimized layers couldn't be shared across all 10 of those "final" images either: "Hey, there's these optimized Ubuntu image layers without the package caches, so we'll use it for our .NET, Java, Node and other images" as opposed to --squash which would put everything in a single large layer, thus removing the benefits from the shared layers of the base Ubuntu image and so on. Who knows, maybe someone will write a tool like that some day.
e12e 4y ago
Note that the explicit "apt-get clean" is probably redundant: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ https://docs.docker.com/develop/develop-images/dockerfile_be... > Official Debian and Ubuntu images automatically run apt-get clean, so explicit invocation is not required.
- yrro 4y ago
  'apt-get clean' doesn't clear out /var/lib/apt/lists. It removes cached downloaded debs from /var/cacpt/apt but you'll still have hundreds of MiB of package lists on your system after running it.
  - e12e 4y ago
    Yes, but apt-get clean is still redundant [ed: because the upstream Debian/Ubuntu images automatically runs apt-clean via how dpkg/apt is configured - and your image should inherit this behavior]. Personally I'm not a fan of deleting random files like man pages and documentation - so instead of: RUN apt-get update -qq && \ apt-get install -y build-essential libvips && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man I'd do: RUN apt-get update -qq && \ apt-get install -y build-essential libvips && \ rm -rf /var/lib/apt/lists/*
    - cvccvroomvroom 4y ago
      Not a fan of violently modifying the system outside of the package manager either. rm -rf isn't configuration management, it's system entropy increasing leaving users scrambling to reinstall the world. If people don't want docs, then distro vendors should package them separately and make them recommended packages.
      - e12e 4y ago
        Indeed. If you want a small image, there's "slim" and "alpine" variants.
  - 4y ago
bradgessler 4y ago
I wanted to explain layers really bad, Sam Ruby even put comments in there about it, but I stayed away from it for the sake of more clearly explaining how Linux was configured for Rails apps. It is really weird looking at Dockerfiles for the first time seeing all of the `&% \` bash commands chained together.
vidarh 4y ago
An terbative to removing files or go through contortions to stuff things in a single layer is to use a builder image and copy the generated artefacts into a clean image: FROM foo AS builder .. build steps FROM foo COPY --from=builder generated-file target (I hope I got that right; on a phone and been a while since I did this from scratch, but you get the overall point)
- d3nj4l 4y ago
  This isn’t really useful for frameworks like rails, since there’s nothing to “compile” there. Most rails docker images will just include the runtime and a few C dependencies, which you need to run the app.
  - vidarh 4y ago
    There are however temporary files being downloaded for the apt installation, and while in this case it's simple enough to remove them in one step that's by no means always the case. Depending on which gems you decide to rely on you may e.g. also end up with a full toolchain to build extensions and the like, so knowing the mechanism is worthwhile.
    - theptip 4y ago
      How would you go about copying something you installed from apt in a build container? Say `apt install build-essential libvips` from the OP, it's not obvious to me what files libvps is adding. I suppose there's probably an incantation for that? What about something that installs a binary? Seems like a pain to chase down everything that's arbitrarily touched by an apt install, am I missing some tooling that would tame that pain?
      - viraptor 4y ago
        Run "dpkg -L libvips" to find the files belonging to that package. This doesn't cover what's changed in post install hooks, but for most docker-relevant things, it's good enough.
JimDabell 4y ago
If all you want to do is run a series of commands while only creating a single layer, heredocs are probably the simplest / most readable approach: https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ https://www.docker.com/blog/introduction-to-heredocs-in-dock...
- e12e 4y ago
  I like here-docs, but frequently I think just making small shell scripts to be invoked by RUN is better, eg putting apt invocations in something like: buildscripts/install_deps and simply RUN that from the Dockerfile.
- tomca32 4y ago
  Why didn't I ever think of this. This is great and really makes things way simpler. Thanks!
  - jbott 4y ago
    Unfortunately this syntax is not generally supported yet - it's only supported with the buildkit backend and only landed in the 1.3 "labs" release. It was moved to stable in early 2022 (see https://github.com/moby/buildkit/issues/2574 https://github.com/moby/buildkit/issues/2574), so that seems to be better, but I think may still require a syntax directive to enable. Many other dockerfile build tools still don't support it, e.g. buildah (see https://github.com/containers/buildah/issues/3474 https://github.com/containers/buildah/issues/3474) Useful now if you have control over the environment your images are being built in, but I'm excited to the future where it's commonplace!
    - Groxx 4y ago
      And AFAIK buildkit is still a real pain to troubleshoot, as you can't use intermediate stages :/ You can put stuff in a script file and just run that script too.
      - cpuguy83 4y ago
freedomben 4y ago
Yeah this is a very widely misunderstood or unknown thing about docker files. After the nth time explaining it to somebody, I finally threw it into a short video with demo to explain how it worked: https://youtu.be/RP-z4dqRTZA https://youtu.be/RP-z4dqRTZA
dclowd9901 4y ago
Such a helpful comment. Just in this one note, I learned a few things about Docker that I had no idea about: 1) a layer is essentially a docker image in itself and 2) a layer is as static as the image itself 3) Docker images ship with all layers Thanks jchw!
Too 4y ago
The new best practice is to use the RUN --mount cache options. Making the removal of intermediate files unnecessary and speeds up the builds too. Surprised to see so few mentions of it.
- skrtskrt 4y ago
  hard enough to stay synced with all the latest docker changes, let alone at an organizational level. Example: in the last few years docker compose files have gone from version 2 to version 3 (missing tons of great v2 features in the name of simplification) to the newest, unnamed unnumbered version which is a merging of versions 2 and 3.
- samruby 4y ago
  At the moment Rails is focused on simplicity/readability. I've got a gem that I'm proposing (and DHH is evaluating) that adds caching as an option: https://github.com/rubys/dockerfile-rails#overview https://github.com/rubys/dockerfile-rails#overview
- zbentley 4y ago
  As someone whose week was ruined by an overwhelming proliferation of `--mount type=cache` options throughout a complex build system, I'm not so sure. Externally managed caches don't have a lifecycle controlled or invalidated by changes in Dockerfiles, or by docker cache-purging commands like "system prune". That means you have to keep track of those caches yourself, which can be a pain in complex, multi-contributor environments with many layers and many builds using the same caches (intentionally or by mistake).
  - TylerE 4y ago
    Could you have the first step of the dockerfile hash itself and blow away the cache(s) if the hash changes?
    - zbentley 4y ago
      Something roughly equivalent is the medium-term solve, yes. Complicated by the fact that this pattern was used by rather a lot of different folks' code.
rendaw 4y ago
Self hoisting here, I put this together to make it easier to generate single (extra) layer docker images without needing a docker daemon, capabilities, chroot, etc: https://github.com/andrewbaxter/dinker https://github.com/andrewbaxter/dinker Caveat: it doesn't work on Fly.io. They seem to be having some issue with OCI manifests: https://github.com/containers/skopeo/issues/1881 https://github.com/containers/skopeo/issues/1881 . They're also having issues with new docker versions pushing from CI: https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/19 https://community.fly.io/t/deploying-to-fly-via-github-actio... ... the timing of this post seems weird. FWIW the article says > create a Docker image, also known as an OCI image I don't think this is quite right. From my investigation, Docker and OCI images are basically content addressed trees, starting with a root manifest that points to other files and their hashes (root -> images -> layers -> layer configs + files). The OCI manifests and configs are separate to Docker manifests and configs and basically Docker will support both side by side.
- cvccvroomvroom 4y ago
  Self-hosting and self-hoisting here. Whenever docker gives me grief, I use podman.
- mrkurt 4y ago
  The newest buildkit shipped a regression. This broke the Docker go client and containerd's pull functionality. It has been a mess: https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/19?u=kurt https://community.fly.io/t/deploying-to-fly-via-github-actio...
sequoia 4y ago
Is there some reason not to use multi-stage builds for this? Blogspam: https://sequoia.makes.software/reducing-docker-image-size-particularly-for-kubernetes-environments/#multi-stage-builds-em-current-approach-use-this-one-em-em-em- https://sequoia.makes.software/reducing-docker-image-size-pa...

by jchw 4y ago

viraptor 4y ago

If you want to check your images for some common leftover files in all the layers, I made an app for that: https://github.com/viraptor/cruftspy https://github.com/viraptor/cruftspy

cvccvroomvroom 4y ago

Every command in a dockerfile creates a layer, and most Dockerfile builders cache each if that line hasn't changed. Dynamic callouts to run updates or check things on the web won't rerun.

mattficke 4y ago

The older “Docker without Docker” blogpost linked from there goes into that, it’s one of the best deep dives into containers (and honestly one of the best pieces of technical writing full stop) I’ve come across. https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/

emerongi 4y ago

If you only care about the final image size, then `docker build --squash` squashes the layers for you as well.

jchw 4y ago
Definitely, although it's worth noting that while the image size will be smaller, it will get rid of the benefits of sharing base layers. Having less redundant layers lets you save most of the space without losing any of the benefits of sharing slices. I think that is the main reason why this is not usually done.
gkfasdfasdf 4y ago
--squash is still experimental, I believe multi-stage images are the new best practice here.
- 411111111111111 4y ago
  in case anyone doesn't know what that means, its basically this kind of dockerfile FROM the_source_image as builder RUN build.sh FROM the_source_image COPY --from=builder /app/artifacts /app/ CMD .... i'm not sure if you can really call it the new best practice though, its been the default for ... a very long time at this point.
  - jchw 4y ago
    Typically I wind up using a different source image for the builder that ideally has (most of) the toolchain bits needed, but the same runtime base as the final image. (For Go, go:alpine and alpine work well. I'm aware alpine/musl is not technically supported by Go, but I have yet to hit issues in prod with it, so I guess I'll keep taking that gamble.)
    - deleted 4y ago
      [deleted]
  - KronisLV 4y ago
    I take advantage of multi-stage builds, however I still think that the layer system could have some nice improvements done to it. For example, say I have my own Ubuntu image that is based on one of the official ones, but adds a bit of common configuration or tools and so on, on which I then build my own Java image using the package manager (not unlike what Bitnami do with their minideb, on which they then base their PostgreSQL and most other container images). So I might have something like the following in the Ubuntu image Dockerfile: RUN apt-get update && apt-get install -y \ curl wget \ net-tools inetutils-ping dnsutils \ supervisor \ && apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/* But then, if I want to install additional software, I need to fetch the package list anew downstream: FROM my-own-repo/ubuntu RUN apt-get update && apt-get install -y \ openjdk-17-jdk-headless \ && apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/* As opposed to being able to just leave the cache files in the previous layers/images, then remove them in a later layer and just do something like: docker build -t my_optimized_java_image -f java.Dockerfile --purge-deleted-files . or maybe docker build -t my_regular_java_image -f java.Dockerfile . purge-deleted-files -t my_regular_java_image -o my_optimized_java_image Which would then work backwards from the last layer and create copies of all of the layers where files have been removed/masked (in the later layers) to use instead of the originals. Thus if I'd have 10 different images that need to use apt to install stuff while building them, I could leave the cache in my own Ubuntu image and then just remove it for whatever I want to consider the "final" images that I'll ship, which would then alter the contents of the included layers to purge deleted files. There's little reason why these optimized layers couldn't be shared across all 10 of those "final" images either: "Hey, there's these optimized Ubuntu image layers without the package caches, so we'll use it for our .NET, Java, Node and other images" as opposed to --squash which would put everything in a single large layer, thus removing the benefits from the shared layers of the base Ubuntu image and so on. Who knows, maybe someone will write a tool like that some day.

e12e 4y ago

Note that the explicit "apt-get clean" is probably redundant: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ https://docs.docker.com/develop/develop-images/dockerfile_be... > Official Debian and Ubuntu images automatically run apt-get clean, so explicit invocation is not required.

yrro 4y ago
'apt-get clean' doesn't clear out /var/lib/apt/lists. It removes cached downloaded debs from /var/cacpt/apt but you'll still have hundreds of MiB of package lists on your system after running it.
- e12e 4y ago
  Yes, but apt-get clean is still redundant [ed: because the upstream Debian/Ubuntu images automatically runs apt-clean via how dpkg/apt is configured - and your image should inherit this behavior]. Personally I'm not a fan of deleting random files like man pages and documentation - so instead of: RUN apt-get update -qq && \ apt-get install -y build-essential libvips && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man I'd do: RUN apt-get update -qq && \ apt-get install -y build-essential libvips && \ rm -rf /var/lib/apt/lists/*
  - cvccvroomvroom 4y ago
    Not a fan of violently modifying the system outside of the package manager either. rm -rf isn't configuration management, it's system entropy increasing leaving users scrambling to reinstall the world. If people don't want docs, then distro vendors should package them separately and make them recommended packages.
    - e12e 4y ago
      Indeed. If you want a small image, there's "slim" and "alpine" variants.
- 4y ago

bradgessler 4y ago

I wanted to explain layers really bad, Sam Ruby even put comments in there about it, but I stayed away from it for the sake of more clearly explaining how Linux was configured for Rails apps. It is really weird looking at Dockerfiles for the first time seeing all of the `&% \` bash commands chained together.

vidarh 4y ago

An terbative to removing files or go through contortions to stuff things in a single layer is to use a builder image and copy the generated artefacts into a clean image: FROM foo AS builder .. build steps FROM foo COPY --from=builder generated-file target (I hope I got that right; on a phone and been a while since I did this from scratch, but you get the overall point)

d3nj4l 4y ago
This isn’t really useful for frameworks like rails, since there’s nothing to “compile” there. Most rails docker images will just include the runtime and a few C dependencies, which you need to run the app.
- vidarh 4y ago
  There are however temporary files being downloaded for the apt installation, and while in this case it's simple enough to remove them in one step that's by no means always the case. Depending on which gems you decide to rely on you may e.g. also end up with a full toolchain to build extensions and the like, so knowing the mechanism is worthwhile.
  - theptip 4y ago
    How would you go about copying something you installed from apt in a build container? Say `apt install build-essential libvips` from the OP, it's not obvious to me what files libvps is adding. I suppose there's probably an incantation for that? What about something that installs a binary? Seems like a pain to chase down everything that's arbitrarily touched by an apt install, am I missing some tooling that would tame that pain?
    - viraptor 4y ago
      Run "dpkg -L libvips" to find the files belonging to that package. This doesn't cover what's changed in post install hooks, but for most docker-relevant things, it's good enough.

JimDabell 4y ago

If all you want to do is run a series of commands while only creating a single layer, heredocs are probably the simplest / most readable approach: https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ https://www.docker.com/blog/introduction-to-heredocs-in-dock...

e12e 4y ago
I like here-docs, but frequently I think just making small shell scripts to be invoked by RUN is better, eg putting apt invocations in something like: buildscripts/install_deps and simply RUN that from the Dockerfile.
tomca32 4y ago
Why didn't I ever think of this. This is great and really makes things way simpler. Thanks!
- jbott 4y ago
  Unfortunately this syntax is not generally supported yet - it's only supported with the buildkit backend and only landed in the 1.3 "labs" release. It was moved to stable in early 2022 (see https://github.com/moby/buildkit/issues/2574 https://github.com/moby/buildkit/issues/2574), so that seems to be better, but I think may still require a syntax directive to enable. Many other dockerfile build tools still don't support it, e.g. buildah (see https://github.com/containers/buildah/issues/3474 https://github.com/containers/buildah/issues/3474) Useful now if you have control over the environment your images are being built in, but I'm excited to the future where it's commonplace!
  - Groxx 4y ago
    And AFAIK buildkit is still a real pain to troubleshoot, as you can't use intermediate stages :/ You can put stuff in a script file and just run that script too.
    - cpuguy83 4y ago

freedomben 4y ago

Yeah this is a very widely misunderstood or unknown thing about docker files. After the nth time explaining it to somebody, I finally threw it into a short video with demo to explain how it worked: https://youtu.be/RP-z4dqRTZA https://youtu.be/RP-z4dqRTZA

dclowd9901 4y ago

Such a helpful comment. Just in this one note, I learned a few things about Docker that I had no idea about: 1) a layer is essentially a docker image in itself and 2) a layer is as static as the image itself 3) Docker images ship with all layers Thanks jchw!

Too 4y ago

The new best practice is to use the RUN --mount cache options. Making the removal of intermediate files unnecessary and speeds up the builds too. Surprised to see so few mentions of it.

skrtskrt 4y ago
hard enough to stay synced with all the latest docker changes, let alone at an organizational level. Example: in the last few years docker compose files have gone from version 2 to version 3 (missing tons of great v2 features in the name of simplification) to the newest, unnamed unnumbered version which is a merging of versions 2 and 3.
samruby 4y ago
At the moment Rails is focused on simplicity/readability. I've got a gem that I'm proposing (and DHH is evaluating) that adds caching as an option: https://github.com/rubys/dockerfile-rails#overview https://github.com/rubys/dockerfile-rails#overview
zbentley 4y ago
As someone whose week was ruined by an overwhelming proliferation of `--mount type=cache` options throughout a complex build system, I'm not so sure. Externally managed caches don't have a lifecycle controlled or invalidated by changes in Dockerfiles, or by docker cache-purging commands like "system prune". That means you have to keep track of those caches yourself, which can be a pain in complex, multi-contributor environments with many layers and many builds using the same caches (intentionally or by mistake).
- TylerE 4y ago
  Could you have the first step of the dockerfile hash itself and blow away the cache(s) if the hash changes?
  - zbentley 4y ago
    Something roughly equivalent is the medium-term solve, yes. Complicated by the fact that this pattern was used by rather a lot of different folks' code.

rendaw 4y ago

Self hoisting here, I put this together to make it easier to generate single (extra) layer docker images without needing a docker daemon, capabilities, chroot, etc: https://github.com/andrewbaxter/dinker https://github.com/andrewbaxter/dinker Caveat: it doesn't work on Fly.io. They seem to be having some issue with OCI manifests: https://github.com/containers/skopeo/issues/1881 https://github.com/containers/skopeo/issues/1881 . They're also having issues with new docker versions pushing from CI: https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/19 https://community.fly.io/t/deploying-to-fly-via-github-actio... ... the timing of this post seems weird. FWIW the article says > create a Docker image, also known as an OCI image I don't think this is quite right. From my investigation, Docker and OCI images are basically content addressed trees, starting with a root manifest that points to other files and their hashes (root -> images -> layers -> layer configs + files). The OCI manifests and configs are separate to Docker manifests and configs and basically Docker will support both side by side.

cvccvroomvroom 4y ago
Self-hosting and self-hoisting here. Whenever docker gives me grief, I use podman.
mrkurt 4y ago
The newest buildkit shipped a regression. This broke the Docker go client and containerd's pull functionality. It has been a mess: https://community.fly.io/t/deploying-to-fly-via-github-action-failing/10171/19?u=kurt https://community.fly.io/t/deploying-to-fly-via-github-actio...

sequoia 4y ago

Is there some reason not to use multi-stage builds for this? Blogspam: https://sequoia.makes.software/reducing-docker-image-size-particularly-for-kubernetes-environments/#multi-stage-builds-em-current-approach-use-this-one-em-em-em- https://sequoia.makes.software/reducing-docker-image-size-pa...