8 ms·

As is the case with the Docker's best practices for Dockerfiles in the official documentation, they're leaving out some really important details. Specifically,

As is the case with the Docker's best practices for Dockerfiles in the official documentation, they're leaving out some really important details. Specifically, they don't really express how Docker packaging is a process integrating the way you build, where you build, and how you build, not just the Dockerfile. 1. Caching is great... but it can also lead to insecure images because you don't get system package updates if you're only ever building off a cached image. Solution: rebuild once a week from scratch. (https://pythonspeed.com/articles/docker-cache-insecure-images/ https://pythonspeed.com/articles/docker-cache-insecure-image...) 2. Multi-stage builds give you smaller images, but if you don't use them right they result in breaking caching completely, destroying all the speed and size benefits you get from layer caching. Solution: you need to tag and push the build-stage images too, and then pull them before the build, if you want caching to work. (Long version, this is a bit tricky to get right: https://pythonspeed.com/articles/faster-multi-stage-builds/ https://pythonspeed.com/articles/faster-multi-stage-builds/)

avip 7y ago
I honestly don't understand this sentiment (other than shamelessly plugging your posts, which is ok). The tradeoff in any packaging mechanism is always "pin everything to the byte" for maximum security and minimum updateability vs. "blindly update to latest without thinking". We normally develop with the second and deploy using the first and docker is no different.
- itamarst 7y ago
  This isn't really about "getting the latest version", it's about "getting the latest security patches for a stable version." The presumption here is that you're running on a stable base image, Debian Stable or Ubuntu LTS or something. Updates to system packages are therefore basically focused on security fixes and severe bug fixes (data corruption in a database, say). Even if you're pinning everything very specifically (and you probably should), you still want to at some point get these (severe-bugfix-only) updates on a regular basis for security and operational reasons. Blind reliance on caching prevents this from happening.
  - geezerjay 7y ago
    > This isn't really about "getting the latest version", it's about "getting the latest security patches for a stable version." Typically security patches trigger new releases with minor/patch version number bumps, which are then installed by getting the latest version of the package. That's pretty much the SOP of any linux distribution.
javagram 7y ago
> Solution: rebuild once a week from scratch. I’ve noticed the official docker images don’t seem to do this. E.g. the official “java” images seem to be uploaded and then are never changed, the only way to get a newer underlying base system is to upgrade to a newer version tag release. Is this true of all the official images, I wonder?
- itamarst 7y ago
  The Python ones seem to be rebuilt much more frequently (last update was 10 days ago). This is perhaps because it depends on pip which has frequent releases.
- koolba 7y ago
  Using tagged upstreams is a good idea as it puts you in control of forcing an upgrade. Best combo is to pin to a specific tag, that you periodically update to the latest stable release, and also allow overriding via a build arg. Anyone who wants the bleeding edge, say for a CI server, can run a build with “latest” as the tag arg.
- colek42 7y ago
  We check once a day to see if the upstream repo has been updated and build our base images. I have used versions of this with clients. https://github.com/boxboat/auto-dockerfile https://github.com/boxboat/auto-dockerfile
- cpuguy83 7y ago
  This is not true. The images are rebuilt automatically when base images are updated.
  - javagram 7y ago
    What is the source for that? When I looked into this before, I wasn't able to find anything in the documentation stating this would happen. Here's the official Node.JS image from a couple years ago, for example... $ sudo docker inspect node:6.1 | grep 'Z"' "Created": "2016-05-06T21:57:54.091444751Z", "LastTagTime": "0001-01-01T00:00:00Z" Node 6.1.0 was released on May 6 2016, it looks to me like the image was never changed after that? And if I run `ls -lah /var/lib/dpkg/info/*.list` inside the image, I get a modification time of May 3, 2016 on all the files... I tried the "node:10.0" image as well and I see similar behavior.
atmosx 7y ago
3. Don't run processes are root, most of the times you don't have to, it's an easy easy win and protects against many docker security CVEs.
- itamarst 7y ago
  Yes! To be fair the official documentation Dockerfile best practices does mention this (although it also gives contradictory advice).
  - cmdkeen 7y ago
    The documentation may do but many of the official images run as root and provide little to no documentation on how to change that, the Bitnami images in comparison are light years ahead.
    - atmosx 7y ago
      A simple multistage example for Golang apps would be: FROM golang:1.12.4 WORKDIR /opt/src/github.com/project1/myprog/ COPY . . RUN go get -d -v ./... RUN CGO_ENABLED=0 GOOS=linux go build -a -mod=vendor -o myprog . FROM ubuntu:latest RUN useradd -u 5002 user1 FROM scratch COPY --from=0 /opt/src/github.com/project1/myprog/myprog . COPY --from=1 /etc/passwd /etc/passwd USER user1 ENTRYPOINT ["./myprog"]
      - cmdkeen 7y ago
        Indeed - and there's no mention of any of this at https://hub.docker.com/_/golang https://hub.docker.com/_/golang - compare with https://hub.docker.com/r/bitnami/nginx https://hub.docker.com/r/bitnami/nginx that discussed non-root users and how to use it.
asdkhadsj 7y ago
I find it odd that docker pipelines don't cache all the layers by default. That is to say, that they don't push all cached layers to a central repository. I'm not even sure if it's possible. Is there a reason why this would be bad? Clearly you'd have to clean the old cache layers regularly, but I'm more concerned with some layers taking very long times - the caching seems required. Though I suppose if you set up your CI system to store the caches locally then you get caching, and it's more efficient as you're not downloading layers. So maybe that's just the "right" way to do it, regardless. /shrug
- itamarst 7y ago
  Sometimes multi-stage builds are used to get build secrets into the build image. The final image just copies the resulting binaries over, and so you don't want to push the build image since you'd be leaking secrets. (I talk about alternatives to that here, since caching the build image is something you do want: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/). Eventually this will be unnecessary given (currently experimental) secret-sharing features Docker is adding. But for now pushing everything by default would be a security risk for some.
  - meowface 7y ago
    Just to add onto your second point for others who might not be aware, the experimental secret and SSH agent forwarding features have greatly simplified a lot of Dockerfiles I work on. SSH forwarding in particular has been really helpful for dealing with private dependencies. There's a good summary here: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 https://medium.com/@tonistiigi/build-secrets-and-ssh-forward.... The tl;dr is you can now write lines like "RUN --mount=type=ssh git clone ssh://git@domain/project.git" (or any command that uses SSH) in a Dockerfile to use your host machine's SSH agent during "docker build". You do currently need to specify experimental syntax in the Dockerfile, and set the DOCKER_BUILDKIT environment variable to 1, and pass "--ssh default" to "docker build", but it's a great workflow improvement IMO.
- cpuguy83 7y ago
davnicwil 7y ago
The flipside of 1. though is that cached images can't be hijacked and have vulnerabilities injected, or from a non security standpoint, have breaking changes introduced which mean the image no longer builds and/or runs how you're expecting.
itamarst 7y ago
I ended up expanding this into a blog post covering not the linked article (though it shares some of the same flaws), but the official docs: https://pythonspeed.com/articles/official-docker-best-practices/ https://pythonspeed.com/articles/official-docker-best-practi...
toniti 7y ago
> Solution: you need to tag and push the build-stage images too With BuildKit cache for all the intermediate stages is tracked. You can push the whole cache with buildx or inline the intermediate cache metadata in buildx or v19.03. `--cache-from` will pull in matched layers automatically on build. Can also export the cache to a shared volume if that suites you better than a registry.

avip 7y ago

I honestly don't understand this sentiment (other than shamelessly plugging your posts, which is ok). The tradeoff in any packaging mechanism is always "pin everything to the byte" for maximum security and minimum updateability vs. "blindly update to latest without thinking". We normally develop with the second and deploy using the first and docker is no different.

itamarst 7y ago
This isn't really about "getting the latest version", it's about "getting the latest security patches for a stable version." The presumption here is that you're running on a stable base image, Debian Stable or Ubuntu LTS or something. Updates to system packages are therefore basically focused on security fixes and severe bug fixes (data corruption in a database, say). Even if you're pinning everything very specifically (and you probably should), you still want to at some point get these (severe-bugfix-only) updates on a regular basis for security and operational reasons. Blind reliance on caching prevents this from happening.
- geezerjay 7y ago
  > This isn't really about "getting the latest version", it's about "getting the latest security patches for a stable version." Typically security patches trigger new releases with minor/patch version number bumps, which are then installed by getting the latest version of the package. That's pretty much the SOP of any linux distribution.

javagram 7y ago

> Solution: rebuild once a week from scratch. I’ve noticed the official docker images don’t seem to do this. E.g. the official “java” images seem to be uploaded and then are never changed, the only way to get a newer underlying base system is to upgrade to a newer version tag release. Is this true of all the official images, I wonder?

itamarst 7y ago
The Python ones seem to be rebuilt much more frequently (last update was 10 days ago). This is perhaps because it depends on pip which has frequent releases.
koolba 7y ago
Using tagged upstreams is a good idea as it puts you in control of forcing an upgrade. Best combo is to pin to a specific tag, that you periodically update to the latest stable release, and also allow overriding via a build arg. Anyone who wants the bleeding edge, say for a CI server, can run a build with “latest” as the tag arg.
colek42 7y ago
We check once a day to see if the upstream repo has been updated and build our base images. I have used versions of this with clients. https://github.com/boxboat/auto-dockerfile https://github.com/boxboat/auto-dockerfile
cpuguy83 7y ago
This is not true. The images are rebuilt automatically when base images are updated.
- javagram 7y ago
  What is the source for that? When I looked into this before, I wasn't able to find anything in the documentation stating this would happen. Here's the official Node.JS image from a couple years ago, for example... $ sudo docker inspect node:6.1 | grep 'Z"' "Created": "2016-05-06T21:57:54.091444751Z", "LastTagTime": "0001-01-01T00:00:00Z" Node 6.1.0 was released on May 6 2016, it looks to me like the image was never changed after that? And if I run `ls -lah /var/lib/dpkg/info/*.list` inside the image, I get a modification time of May 3, 2016 on all the files... I tried the "node:10.0" image as well and I see similar behavior.

atmosx 7y ago

3. Don't run processes are root, most of the times you don't have to, it's an easy easy win and protects against many docker security CVEs.

itamarst 7y ago
Yes! To be fair the official documentation Dockerfile best practices does mention this (although it also gives contradictory advice).
- cmdkeen 7y ago
  The documentation may do but many of the official images run as root and provide little to no documentation on how to change that, the Bitnami images in comparison are light years ahead.
  - atmosx 7y ago
    A simple multistage example for Golang apps would be: FROM golang:1.12.4 WORKDIR /opt/src/github.com/project1/myprog/ COPY . . RUN go get -d -v ./... RUN CGO_ENABLED=0 GOOS=linux go build -a -mod=vendor -o myprog . FROM ubuntu:latest RUN useradd -u 5002 user1 FROM scratch COPY --from=0 /opt/src/github.com/project1/myprog/myprog . COPY --from=1 /etc/passwd /etc/passwd USER user1 ENTRYPOINT ["./myprog"]
    - cmdkeen 7y ago
      Indeed - and there's no mention of any of this at https://hub.docker.com/_/golang https://hub.docker.com/_/golang - compare with https://hub.docker.com/r/bitnami/nginx https://hub.docker.com/r/bitnami/nginx that discussed non-root users and how to use it.

asdkhadsj 7y ago

I find it odd that docker pipelines don't cache all the layers by default. That is to say, that they don't push all cached layers to a central repository. I'm not even sure if it's possible. Is there a reason why this would be bad? Clearly you'd have to clean the old cache layers regularly, but I'm more concerned with some layers taking very long times - the caching seems required. Though I suppose if you set up your CI system to store the caches locally then you get caching, and it's more efficient as you're not downloading layers. So maybe that's just the "right" way to do it, regardless. /shrug

itamarst 7y ago
Sometimes multi-stage builds are used to get build secrets into the build image. The final image just copies the resulting binaries over, and so you don't want to push the build image since you'd be leaking secrets. (I talk about alternatives to that here, since caching the build image is something you do want: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/). Eventually this will be unnecessary given (currently experimental) secret-sharing features Docker is adding. But for now pushing everything by default would be a security risk for some.
- meowface 7y ago
  Just to add onto your second point for others who might not be aware, the experimental secret and SSH agent forwarding features have greatly simplified a lot of Dockerfiles I work on. SSH forwarding in particular has been really helpful for dealing with private dependencies. There's a good summary here: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 https://medium.com/@tonistiigi/build-secrets-and-ssh-forward.... The tl;dr is you can now write lines like "RUN --mount=type=ssh git clone ssh://git@domain/project.git" (or any command that uses SSH) in a Dockerfile to use your host machine's SSH agent during "docker build". You do currently need to specify experimental syntax in the Dockerfile, and set the DOCKER_BUILDKIT environment variable to 1, and pass "--ssh default" to "docker build", but it's a great workflow improvement IMO.
cpuguy83 7y ago

davnicwil 7y ago

The flipside of 1. though is that cached images can't be hijacked and have vulnerabilities injected, or from a non security standpoint, have breaking changes introduced which mean the image no longer builds and/or runs how you're expecting.

itamarst 7y ago

I ended up expanding this into a blog post covering not the linked article (though it shares some of the same flaws), but the official docs: https://pythonspeed.com/articles/official-docker-best-practices/ https://pythonspeed.com/articles/official-docker-best-practi...

toniti 7y ago

> Solution: you need to tag and push the build-stage images too With BuildKit cache for all the intermediate stages is tracked. You can push the whole cache with buildx or inline the intermediate cache metadata in buildx or v19.03. `--cache-from` will pull in matched layers automatically on build. Can also export the cache to a shared volume if that suites you better than a registry.