12 ms·
`COPY –chmod` reduced the size of my container image by 35%
- 3np 5y agoThis and other common considerations and gotchas are brought up in Shekow's excellent Docker optimization guide. https://www.augmentedmind.de/2022/02/06/optimize-docker-image-size/ https://www.augmentedmind.de/2022/02/06/optimize-docker-imag... Was posted here last month: https://news.ycombinator.com/item?id=30406076 https://news.ycombinator.com/item?id=30406076
- dmix 5y agoI noticed you start to find every operating system quirk ever when you start writing Dockerfiles. I've run into so many strange things just converting a simple predictable shell script into a Dockerfile.
- _jal 5y agoWelcome to the 'ops' in devops.
- travisd 5y ago“Welcome! Sorry.”
- speed_spread 5y agoMore like DevOops
- viraptor 5y agoTbh, that's just a docker quirk. There were so many ways to do this with a more reasonable implementation and yet...
- dmix 5y agoI initially just wantetd to replicate my production env locally, then after that experience I wanted to replicate my Docker env in production so that would also be predictable.
- Too 5y agoWhy not run your existing predictable script in a single RUN command? What need to be converted? The mistake many do is seeing dockerfiles as a 1:1 mapping of a shell script with RUN prefixed on every line. It’s not, you should only split a run if you have a good reason to add a new COPY in between for layer caching reasons or switching user. With the --bind options from buildkit you can ensure the apt cache does not get layered and you can mount any big temporary files needed from host instead of copying them first.
- rmetzler 5y agoAlso each RUN, COPY, ADD, etc. creates a new layer in the image so you should put commands which are related in the same command. The example in the blog post violates this principle, which is the main reason for the unexpected size.
- fivea 5y ago> I noticed you start to find every operating system quirk ever when you start writing Dockerfiles. I've been using Dockerfiles extensively for years and I'm yet to find anything that fits the definition of a OS quirk. The quirkiest thing I've noticed in Dockerfiles is the ADD vs COPY thing. > I've run into so many strange things just converting a simple predictable shell script into a Dockerfile. What exactly are you trying to do setting up a Dockerfile that requires a full blown shell script? A Dockerfile should have little more beyond updating/installing system packages with a package manager, and copying files into the container image. First you run a build to get your artifacts ready for packaging, and afterwards you package those artifacts by running your Dockerfile.
- ghoulishboo 5y ago
- staticassertion 5y ago> I’m kind of impressed with how aggressively this comment misunderstands its parent and then beats the hell out of that misunderstanding. This is just peak irony.
- ghoulishboo2 5y ago
- staticassertion 5y agoYou learned what irony was in a literary theory class?
- TobTobXX 5y agoI don't know what qualifues a s quirk, but I sometimes had to think about stuff like "who's PID 0", "do I have the CAP_WHATEVER capability", "I need to 'reap' subprocesses?" and other stuff that 'just works' when you have a decent system with a decent init process and all the other things.
- nodesocket 5y agoA very common mistake I see (though not related to image size perse) when running Node apps is to do CMD ["npm", "run", "start"]. This is first memory wasteful, as npm is running as the parent process and forking node to run the main script. Also, the bigger problem is that the npm process does not send signals down to its child thus SIGINT and SIGTERM are not passed from npm into node which means your server may not be gracefully closing connections.
- arnaudsm 5y agoWhat do you recommend instead?
- nodesocket 5y agoJust invoke your script: CMD ["/usr/local/bin/node", "server.js"]
- TameAntelope 5y agopm2 is good for some things.
- nodesocket 5y agopm2 is great when running on servers, but using pm2 in containers feels wrong and again wasteful. Just invoke your script. If it crashes, fine Kubernetes or Docker handles that. Logs, handled by k8s. Monitoring I use DataDog.
- chousuke 5y agoWhat's the point of pm2? Every time I've seen it it's just been part of a messy misconfigured system and whatever it's actually doing could've been accomplished entirely with a tiny systemd unit running node directly.
- TameAntelope 5y ago
- oefrha 5y agoThis problem can also be solved through squashing the layers, which is a lot more general solution.
- dpedu 5y agoWhat's a good tool for doing this? I assume that you don't want to merge your custom layers with the base image layer.
- FooBarWidget 5y agoAnd squashing is still behind experimental flag after all these years.
- encryptluks2 5y agoNot with Buildah
- FooBarWidget 5y agoWhich doesn't work on macOS. And which has no caching.
- encryptluks2 5y agoI think it is built into the podman build code as well.
- nhoughto 5y agoDo it all in Userland with jib cli, doesn’t work for all cases but it’s constraints generally keep you honest.
- FooBarWidget 5y agoThat only works for Java. Nearly all of the stuff I work on involves native executables as well as the need to setup the OS environment in the container (libraries, user accounts, etc)
- dpedu 5y agoI love docker, but it is baffling that the obvious need for a COPY or ADD argument like this has not been satisfied yet.
- habitue 5y agoSo many gotchas like this in dockerfiles. I think the issue stems from it being such a leaky abstraction. To use it correctly you need to know how docker works internally inside and out, as well as Linux inside and out. The default choices are baffling in docker, it really is a worse-is-better kind of tool. Has anyone worked on a replacement for dockerfiles? I know buildah is an alternative to docker build, but it just uses the same file format
- fenollp 5y agohttps://github.com/moby/buildkit https://github.com/moby/buildkit
- viraptor 5y agoYou can also use buildah commands without the whole dockerfile abstraction. As a structured alternative there's also an option to build container images from nix expressions.
- Too 5y agoNever tried myself but this is the most serious attempt I’ve seen on alternative docker syntax https://earthly.dev/ https://earthly.dev/ You also have mockerfiles, being more of a proof of concept if I understand correctly https://matt-rickard.com/building-a-new-dockerfile-frontend/ https://matt-rickard.com/building-a-new-dockerfile-frontend/
- adamgordonbell 5y agoEarthly is great (disclosure: work on it) But also checkout out IckFiles, an Intercal frontend for moby buildkit: https://github.com/adamgordonbell/compiling-containers/tree/main/ickfile https://github.com/adamgordonbell/compiling-containers/tree/...
- xinnixxinix 5y agoSure, there are, but they all have enough of a learning curve that they don't seem to take hold with the masses. Nix, Guix, Bazel, Habit, and others, all solve this problem more elegantly. There are some big folks out there, quiet quietly using Nix to solve: * reproducible builds * shared remote/CI builds * trivial cross-arch support * minimal container images * complete knowledge of all SW dependencies and what-is-live-where * "image" signing and verification I know docker and k8s well and it's kind of silly how much simpler the stack could be made if even 1% of the effort spent working around Docker were spent by folks investing in tools that are principally sound instead of just looking easy at first glance. Miss me with the complaints about syntax. It's just like Rust. Any pain of learning is very quickly forgotten by the unbridled pace at which you can move. And besides, it's nothing compared to (looks at calendar) 5 years of "Top 10 Docker Pitfalls!" as everyone tries to pretend the teetering pile of Go is making their tech debt go away. I never thought I'd come around to being someone wary of the word "container", as someone who sorta made it betting on them. There is so little care for actually managing and understanding the depth of one's software stack, well, we have this. (Pouring one out for yet another Dockerfile with apt-get commands in it.)
- osener 5y agoIn my experience docker-slim[0] is the way to go for creating minimal and secure Docker images. I wasted a lot of time in the past trying to ship with Alpine base images and statically compiling complicated software. All the performance, compatibility, package availability headaches this brings is not worth it when docker-slim does a better job of removing OS from your images while letting you use any base image you want. Tradeoff is that you give up image layering to some extent and it might take a while to get dead-file-elimination exactly right if your software loads a lot of files dynamically (you can instruct docker-slim to include certain paths and probe your executable during build). If docker-slim is not your thing, “distroless” base images [1] are also pretty good. You can do your build with the same distro and then in a multi stage docker image copy the artifacts into distroless base images. [0] https://github.com/docker-slim/docker-slim https://github.com/docker-slim/docker-slim [1] https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
- kylequest 5y agoLayers support in docker-slim is something that's on the todo list (exact time is tbd). The recent set of the engine enhancements reduce the need for the explicit include flags. Also new application-specific include capabilities are wip (already added a number of new flags for node.js, next.js and nuxt.js applications).
- AtNightWeCode 5y agoI have quit using Alpine. It caused to many issues in production. For some workloads in GO for instance you can use Scratch directly. But slim and distroless are my preferred base images.
- marsven_422 5y agoYou have a problem, you use docker to solve it, now you have two problems.
- georgia_peach 5y agoStrikingly more accurate than the original regex quote. They believe it was Wilde who said, “If you want to tell people the truth, you’d better make them laugh or they’ll kill you.”
- dhruvrrp 5y agoSlight nitpick, but `apt-get update && apt-get install -y openssl dumb-init iproute2 ca-certificates` in the dockerfile is not the recommended approach. That command itself means that a docker container is no longer reproducible. You cannot build it (with any code changes for your service) and guaranteed to be the same since that might be in production due to changes in the packages. Always better to go with the base image, add your packages to the base and then use that new image as the base image for your application.
- wichert 5y agoAt that point your base image is not reproducible, so your improvement is going to be very limited.
- fivea 5y ago> That command itself means that a docker container is no longer reproducible. It's a tradeoff between making container images reproducible, and not shipping security vulnerabilities. People tend to prefer the latter. Furthermore, you can exec your way into a container and check exactly which package version you installed.
- hericium 5y ago> It's a tradeoff between making container images reproducible, and not shipping security vulnerabilities. You can regenerate your base images every day or more often and have consistent containers created from an image. Freshly generated image can be tested in a pipeline to avoid issues and you won't hit issues like inability to scale due to misbehaving new containers.
- maxekman 5y agoI would recommend Google Ko if you are packaging Go apps: https://github.com/google/ko https://github.com/google/ko
- lenn0x 5y agoCame here to suggest just this. Ever since coming across ko, it's been excellent in our CI pipelines.
- spullara 5y agoEveryone is talking about workarounds when this should be fixed in the file system. This is just dumb. Changing metadata shouldn't require the entire file to be copied lol.
- Hendrikto 5y ago> If you are wondering why a metadata update would make OverlayFS duplicate the entire file, it is for security reasons. You can enable “metadata only copy up”[0] feature which will only copy the metadata instead of the whole file. [0]: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#metadata-only-copy-up https://www.kernel.org/doc/html/latest/filesystems/overlayfs...
- benreesman 5y agoYou folks do all realize that almost all of this is trying to work around the fact that Ulrich Drepper is cramming dynamically-linked glibc up our uh, software stack, right? Linus doesn’t break userland. A tarball is a deployment strategy if someone isn’t dicking with /usr/lib under you.
- maccolgan 5y agoI still use NSS.
- dark-star 5y agoThis shouldn't apply if you use btrfs as backend filesystem though, does it?
- TOGoS 5y agoMy understanding (based on other comments in the thread - I'm no Docker internals expert) is that it's about the size of Docker image files, which contain a tarball (or similar) of files that each layer adds or modifies on top of its base layer. There's no way for them to say "same as this other file, just with permissions changed". Which has always seemed to me like a bad design decision on Docker's part, because there's lots of room for deduplication within the images that just cannot be done due to the format they chose. Why not have the layers reference individual file by content hash + metadata? If there's a lot of small, unusual files, you could just bundle them with the image, sort of like how Git packs objects together for efficiency, but still retains the identity of each.
- deleted 5y ago[deleted]
- polote 5y agoA better title would be: using chmod doubled the size of my docker container image
- kkfx 5y agoClassic deploy with system-provided "insulation" (GNU/Linux cgroups (firejail/bublewrap) or FreeBSD capsicum etc) reduce far more the size and the overhead...
- hddqsb 5y ago`COPY --chmod` is quite new, and as mentioned in the post requires BuildKit (i.e. `docker buildx`). A more portable solution is to use `chmod` right after unzipping the binary. The `COPY` command will then preserve the executable permission.
- dicknuckle 5y agoReally thought I missed something with COPY --chmod so I'm glad you mentioned it's new. COPY's preservation of flags simplified one of projects because I started setting flags on shell scripts in the repo instead of at build time.
- riccardomc 5y agoFunny, I had a very similar experience at a client of mine last month. They were using Apache Spark images and installing all kind of python libraries on top of them. The biggest contributors to image size were: - miniconda (~2GB) - a final RUN chown -R statement (~750GB) We reduced the image size and relative Spark cluster considerably by playing around with dependencies in order to stick with plain pip and using COPY --chown. I also recommend [dive](https://github.com/wagoodman/dive https://github.com/wagoodman/dive) analyse what contributes to each layer.
- wabain 5y agoI think the OP is confusing the runtime and image format a bit here. At runtime OverlayFS can use metadata-only copy up to describe changed files, but the container image is still defined as a sequence of layers where each layer is a tar file. There's no special handling for metadata-only changes of a file from a parent layer. As the OCI image spec puts it [1]: > Additions and Modifications are represented the same in the changeset tar archive. [1]: https://github.com/opencontainers/image-spec/blob/02efb9a75ee11e05937b535cc5f228f9343ab2f5/layer.md#change-types https://github.com/opencontainers/image-spec/blob/02efb9a75e...
- softwarebeware 5y agoThis site has a nice theme. I wish more of the internet looked like text files.
- 2OEH8eoCRo0 5y agoI found this to be useful as somebody new to containers in general. It's about building from scratch using buildah. https://fedoramagazine.org/build-smaller-containers/ https://fedoramagazine.org/build-smaller-containers/
- googleantitrust 5y agoWhat is the point of these technology inventiones? The desire of hoarding more bananas than the monkey can eat will make this planet unhabitable one day.
- toniti 5y agoUsing `COPY --chmod` is not the correct solution for this. It works, of course, but it isn't very logical from Dockerfile readability standpoint. The real issue is the incorrect use of multi-stage builds. In multi-stage builds you define additional build stages where you prepare your binaries(eg. compiling them) and copy to the final runtime stage, so your final stage remains clean of temporary files created by your build steps. Based on your comment in your current build stage you run curl, extract etc., but you don't actually finish preparing the binary by correcting the executable bit. Instead, you copy the half-prepared binary to runtime stage and then try to continue your further modifications there. Eg. similarily if you would skip extracting step, copy the zip instead and extract it in runtime stage and then you would have the zip and the final binary in your exported image. Another red flag is that you run `apt-get` after copying the binary to runtime stage(because you still want to tweak the binary there). That means any time source for binary changes, the `apt` commands need to run again and are not cached. If you just add the executable bit in your build stage you can reorder them, so the `COPY` comes after `RUN`.
- vamc19 5y agoYou are correct - I should be running chmod in the download stage and that is what I did before realizing `--chmod` existed. However, `--chmod` is still a valid solution. The reason I did not stop with running chmod in the first stage is because this seemed like a common problem - what if I was ADDing a binary or a shell script directly from a remote source and I did not have a download stage? I'm sure there are better ways to write that Dockerfile - I'm by no means an expert. It just so happens that I noticed this problem when the Dockerfile (it was from a different project. I was modifying it) was in this state and I had nothing better to do than ~yak shave~ investigate why the image size was a bit larger than I expected :)
- vermaden 5y agoNo need for such superstition actions on FreeBSD Jails containers :)