6 ms·

Reducing Docker Image Size

176 points by aberoham 7y ago

yjftsjthsd-h 7y ago
Seems to be a pretty decent overview; covers the usual suspects (multi-stage builds, FROM scratch, non-scratch minmal images, ldd to check libraries), with some nice bits that I'd not seen before (busybox:glibc). I would be curious to see how these base images stack up against Google's "distroless" base images (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless). I also appreciate that they call out Alpine's compatibility issues (on account of musl) but still leave it as a thing that can be good if you use it right. (Personally I'm quite fond of Alpine, but I don't bother when using binaries that expect glibc.)
- yegle 7y ago
  One important thing missing from BusyBox:glibc is the SSL certificates that's almost always needed if you are writing things like crawler etc. As for Alpine, at least for packaging Python code, avoid as much as possible. It cannot reuse manylinux packages on PyPI and have to recompile C modules.
  - gopalv 7y ago
    With Alpine, the size advantages of musl sometimes come with hidden overheads as well. Musl's malloc() seems to be a fair bit slower than glibc's when you start to throw in more threads at it (fragments faster?). And then if you try to replace that with jemalloc, you'll quickly find out that jemalloc does not actually get used. At some level, the difference between a 1Mb docker image and a 9Mb one isn't that significant (& I asked for some things like strace, ping and nslookup to be in it, because it gets impossible to debug the container when it obviously caches a DNS record or at least, looks like it is doing that).
  - mxschumacher 7y ago
    and as a result: plenty of breaking builds https://github.com/pyca/cryptography/issues/4264 https://github.com/pyca/cryptography/issues/4264
- fiddlerwoaroof 7y ago
  One thing they don't talk about is using non-docker tools to build images. e.g. nix: http://lethalman.blogspot.com/2016/04/cheap-docker-images-with-nix_15.html http://lethalman.blogspot.com/2016/04/cheap-docker-images-wi... or jib: https://github.com/GoogleContainerTools/jib https://github.com/GoogleContainerTools/jib These may not fit every use case, but they can minimize the size of the image by making really precisely defined containers.
  - donmcronald 7y ago
    It’s worth looking at an overview of how Jib works because the general strategy is very good IMO. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/
- DyslexicAtheist 7y ago
  been using docker-slim[1] for a couple years and am really happy with it. Also I build everything from source and try to avoid glibc wherever I can, use hardened malloc and -D_FORTIFY_SOURCE=2 everything. also use static linking wherever I can and try to --disable-foo with configure if I can get away with it. Using a minimalistic approach that restricts everything (down to a syscall whitelist) sounds like it's high maintenence but when done in combination with a catalog of what runs in these containers and some plumbing to poll upstream for repo updates allows me to ignore a lot of what affects most people wrt security. As a bonus I learn about it quickly if upstream introduces a major change and I'm forced to study the source and get to decide if they were "out of their mind again" with these changes. Keeps me abreast of how my platform behaves and when it doesn't as it should (it's kind of a forward investment for efficient root cause analysis that beats "have you tried turning it off and on again") Docker imo doesn't deserve the hype. Most of the flaws I deal with in my setup are due to docker instability (often in docker swarm) ... it's the next thing I actually want to replace because it's just overhyped "static linking for Millennials". [1] https://github.com/docker-slim/docker-slim https://github.com/docker-slim/docker-slim
  - Jonnax 7y ago
    Docker Swarm has been pretty much deprecated. When Mirantis bought it, they said they'll support for 2 years but Kubernetes is their focus.
    - DyslexicAtheist 7y ago
      thanks, a good reason to ditch it then.
danielhlockard 7y ago
It's worth noting that golang builds can be smaller than that with `GOOS=linux go build -ldflags="-s -w" .` (assuming a build on macos for linux.) From there I usually run `upx --ultra-brute -9 program` before dropping it into a `scratch` docker container (plus whatever other deps it needs).
- zxcmx 7y ago
  Does the UPX help much on top of Docker's layer compression? I can imagine it might (special purpose vs general purpose) but haven't tried it.
  - danielhlockard 7y ago
    Just saw this reply, yes it does.
thinkingemote 7y ago
Why is image size important? Should we instead optimise for speed of build? If storage is cheap. And CPU costs Co2 does it make sense to spend longer time and more energy to save disk space?
- Havoc 7y ago
  >Why is image size important? Cold start times I'd imagine
- dcherman 7y ago
  It's not just the raw size of the image, but also about what the image includes; a smaller image often reduces the potential attack surface because vulnerable things just aren't there. That's one of the major rationales behind the distroless images. Being space optimized is just a really nice side effect.
  - carterehsmith 7y ago
    >> a smaller image often reduces the potential attack surface because vulnerable things just aren't there By the way, the article proposes blind download of artifacts from someplace on the internet, on every build. Not only that can cripple your builds when the source is down (which happens all the time), it can (and that has happened) send you arbitrary infected crap instead of what you wanted.
- tehlike 7y ago
  There are plenty of resource limited use cases, where storage is not exactly cheap. Or updating image over network might be slow or expensive (think edge, over 3-4g)
- hinkley 7y ago
  How many servers are you copying the image to?
- DyslexicAtheist 7y ago
  security (e.g. attack surface)
paxys 7y ago
While such articles are usually helpful, I'd caution that making individual image sizes as small as possible shouldn't really be your goal. As a simple example - if you have 100 unique images in your system, having an image size of 1 GB each where 99% of it is derived from a common layer is going to be a lot smaller overall than "optimizing" the size down to 100 MB each but taking away the base layers.
- leipert 7y ago
  In my experience you 100 unique images might still have different base images. So you end up with too many layers. Especially with interpreted languages those python, ruby or node runtimes add up quickly Also depending on usecase (e.g running a CI system) you might end up with high storage costs (caching) or high bandwidth costs (no caching) Depending where your datacenter is, coworkers with slow connections might suffer as well. Hey, let’s just download 12GB of my everylanguage-and-Tex:latest every workday. But I agree, docker golfing to have minimal images is not always needed, but don’t let your images grow like a wildfire. They are part of your architecture.
  - jacques_chester 7y ago
    Using a trendy small base image doesn't make the intermediate layers any smaller. What you're describing is what cloud native buildpacks are intended to solve: sensible, efficient, auto-updateable OCI images. I worked on them for a while and it just hardened my heart against Dockerfiles.
    - cpitman 7y ago
      It's starting to feel like this is a lost battle though. People have decided that Dockerfiles are "the kubernetes way", even though it exposes developers to details they just shouldn't need to care about. Add anything on top, and it is no longer "vanilla kubernetes". Never mind that means everyone roles their own Rube Goldberg build process, and somehow that is ok...
      - 7y ago
avip 7y ago
This is indeed a good overview. It would have been a great overview had it started with briefing readers about why (or when) image size should bother us at all.
jiofih 7y ago
Strange advice to not use Alpine, but then recommends an empty image or busybox...
- yjftsjthsd-h 7y ago
  I'm pretty sure that's 100% a glibc/musl thing.
snicker7 7y ago
Are containers really the best abstraction for providing application isolation / portability? What about snaps/nix/guix instead?
- yjftsjthsd-h 7y ago
  They're probably not best at any of portability, isolation, or security (they probably lose to full VMs on all counts). But they're good enough and way more convenient / easier to use.

176 points by aberoham 7y ago

Seems to be a pretty decent overview; covers the usual suspects (multi-stage builds, FROM scratch, non-scratch minmal images, ldd to check libraries), with some nice bits that I'd not seen before (busybox:glibc). I would be curious to see how these base images stack up against Google's "distroless" base images (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless). I also appreciate that they call out Alpine's compatibility issues (on account of musl) but still leave it as a thing that can be good if you use it right. (Personally I'm quite fond of Alpine, but I don't bother when using binaries that expect glibc.)

yegle 7y ago
One important thing missing from BusyBox:glibc is the SSL certificates that's almost always needed if you are writing things like crawler etc. As for Alpine, at least for packaging Python code, avoid as much as possible. It cannot reuse manylinux packages on PyPI and have to recompile C modules.
- gopalv 7y ago
  With Alpine, the size advantages of musl sometimes come with hidden overheads as well. Musl's malloc() seems to be a fair bit slower than glibc's when you start to throw in more threads at it (fragments faster?). And then if you try to replace that with jemalloc, you'll quickly find out that jemalloc does not actually get used. At some level, the difference between a 1Mb docker image and a 9Mb one isn't that significant (& I asked for some things like strace, ping and nslookup to be in it, because it gets impossible to debug the container when it obviously caches a DNS record or at least, looks like it is doing that).
- mxschumacher 7y ago
  and as a result: plenty of breaking builds https://github.com/pyca/cryptography/issues/4264 https://github.com/pyca/cryptography/issues/4264
fiddlerwoaroof 7y ago
One thing they don't talk about is using non-docker tools to build images. e.g. nix: http://lethalman.blogspot.com/2016/04/cheap-docker-images-with-nix_15.html http://lethalman.blogspot.com/2016/04/cheap-docker-images-wi... or jib: https://github.com/GoogleContainerTools/jib https://github.com/GoogleContainerTools/jib These may not fit every use case, but they can minimize the size of the image by making really precisely defined containers.
- donmcronald 7y ago
  It’s worth looking at an overview of how Jib works because the general strategy is very good IMO. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/
DyslexicAtheist 7y ago
been using docker-slim[1] for a couple years and am really happy with it. Also I build everything from source and try to avoid glibc wherever I can, use hardened malloc and -D_FORTIFY_SOURCE=2 everything. also use static linking wherever I can and try to --disable-foo with configure if I can get away with it. Using a minimalistic approach that restricts everything (down to a syscall whitelist) sounds like it's high maintenence but when done in combination with a catalog of what runs in these containers and some plumbing to poll upstream for repo updates allows me to ignore a lot of what affects most people wrt security. As a bonus I learn about it quickly if upstream introduces a major change and I'm forced to study the source and get to decide if they were "out of their mind again" with these changes. Keeps me abreast of how my platform behaves and when it doesn't as it should (it's kind of a forward investment for efficient root cause analysis that beats "have you tried turning it off and on again") Docker imo doesn't deserve the hype. Most of the flaws I deal with in my setup are due to docker instability (often in docker swarm) ... it's the next thing I actually want to replace because it's just overhyped "static linking for Millennials". [1] https://github.com/docker-slim/docker-slim https://github.com/docker-slim/docker-slim
- Jonnax 7y ago
  Docker Swarm has been pretty much deprecated. When Mirantis bought it, they said they'll support for 2 years but Kubernetes is their focus.
  - DyslexicAtheist 7y ago
    thanks, a good reason to ditch it then.

danielhlockard 7y ago

It's worth noting that golang builds can be smaller than that with `GOOS=linux go build -ldflags="-s -w" .` (assuming a build on macos for linux.) From there I usually run `upx --ultra-brute -9 program` before dropping it into a `scratch` docker container (plus whatever other deps it needs).

zxcmx 7y ago
Does the UPX help much on top of Docker's layer compression? I can imagine it might (special purpose vs general purpose) but haven't tried it.
- danielhlockard 7y ago
  Just saw this reply, yes it does.

thinkingemote 7y ago

Why is image size important? Should we instead optimise for speed of build? If storage is cheap. And CPU costs Co2 does it make sense to spend longer time and more energy to save disk space?

Havoc 7y ago
>Why is image size important? Cold start times I'd imagine
dcherman 7y ago
It's not just the raw size of the image, but also about what the image includes; a smaller image often reduces the potential attack surface because vulnerable things just aren't there. That's one of the major rationales behind the distroless images. Being space optimized is just a really nice side effect.
- carterehsmith 7y ago
  >> a smaller image often reduces the potential attack surface because vulnerable things just aren't there By the way, the article proposes blind download of artifacts from someplace on the internet, on every build. Not only that can cripple your builds when the source is down (which happens all the time), it can (and that has happened) send you arbitrary infected crap instead of what you wanted.
tehlike 7y ago
There are plenty of resource limited use cases, where storage is not exactly cheap. Or updating image over network might be slow or expensive (think edge, over 3-4g)
hinkley 7y ago
How many servers are you copying the image to?
DyslexicAtheist 7y ago
security (e.g. attack surface)

paxys 7y ago

While such articles are usually helpful, I'd caution that making individual image sizes as small as possible shouldn't really be your goal. As a simple example - if you have 100 unique images in your system, having an image size of 1 GB each where 99% of it is derived from a common layer is going to be a lot smaller overall than "optimizing" the size down to 100 MB each but taking away the base layers.

leipert 7y ago
In my experience you 100 unique images might still have different base images. So you end up with too many layers. Especially with interpreted languages those python, ruby or node runtimes add up quickly Also depending on usecase (e.g running a CI system) you might end up with high storage costs (caching) or high bandwidth costs (no caching) Depending where your datacenter is, coworkers with slow connections might suffer as well. Hey, let’s just download 12GB of my everylanguage-and-Tex:latest every workday. But I agree, docker golfing to have minimal images is not always needed, but don’t let your images grow like a wildfire. They are part of your architecture.
- jacques_chester 7y ago
  Using a trendy small base image doesn't make the intermediate layers any smaller. What you're describing is what cloud native buildpacks are intended to solve: sensible, efficient, auto-updateable OCI images. I worked on them for a while and it just hardened my heart against Dockerfiles.
  - cpitman 7y ago
    It's starting to feel like this is a lost battle though. People have decided that Dockerfiles are "the kubernetes way", even though it exposes developers to details they just shouldn't need to care about. Add anything on top, and it is no longer "vanilla kubernetes". Never mind that means everyone roles their own Rube Goldberg build process, and somehow that is ok...
    - 7y ago

avip 7y ago

This is indeed a good overview. It would have been a great overview had it started with briefing readers about why (or when) image size should bother us at all.

jiofih 7y ago

Strange advice to not use Alpine, but then recommends an empty image or busybox...

yjftsjthsd-h 7y ago
I'm pretty sure that's 100% a glibc/musl thing.

snicker7 7y ago

Are containers really the best abstraction for providing application isolation / portability? What about snaps/nix/guix instead?

yjftsjthsd-h 7y ago
They're probably not best at any of portability, isolation, or security (they probably lose to full VMs on all counts). But they're good enough and way more convenient / easier to use.