4 ms·

I find it odd that docker pipelines don't cache all the layers by default. That is to say, that they don't push all cached layers to a central repository. I'm n

I find it odd that docker pipelines don't cache all the layers by default. That is to say, that they don't push all cached layers to a central repository. I'm not even sure if it's possible. Is there a reason why this would be bad? Clearly you'd have to clean the old cache layers regularly, but I'm more concerned with some layers taking very long times - the caching seems required. Though I suppose if you set up your CI system to store the caches locally then you get caching, and it's more efficient as you're not downloading layers. So maybe that's just the "right" way to do it, regardless. /shrug

itamarst 7y ago
Sometimes multi-stage builds are used to get build secrets into the build image. The final image just copies the resulting binaries over, and so you don't want to push the build image since you'd be leaking secrets. (I talk about alternatives to that here, since caching the build image is something you do want: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/). Eventually this will be unnecessary given (currently experimental) secret-sharing features Docker is adding. But for now pushing everything by default would be a security risk for some.
- meowface 7y ago
  Just to add onto your second point for others who might not be aware, the experimental secret and SSH agent forwarding features have greatly simplified a lot of Dockerfiles I work on. SSH forwarding in particular has been really helpful for dealing with private dependencies. There's a good summary here: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 https://medium.com/@tonistiigi/build-secrets-and-ssh-forward.... The tl;dr is you can now write lines like "RUN --mount=type=ssh git clone ssh://git@domain/project.git" (or any command that uses SSH) in a Dockerfile to use your host machine's SSH agent during "docker build". You do currently need to specify experimental syntax in the Dockerfile, and set the DOCKER_BUILDKIT environment variable to 1, and pass "--ssh default" to "docker build", but it's a great workflow improvement IMO.
cpuguy83 7y ago
Buildkit has some interesting things around storing build cache information in an image.

itamarst 7y ago

Sometimes multi-stage builds are used to get build secrets into the build image. The final image just copies the resulting binaries over, and so you don't want to push the build image since you'd be leaking secrets. (I talk about alternatives to that here, since caching the build image is something you do want: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/). Eventually this will be unnecessary given (currently experimental) secret-sharing features Docker is adding. But for now pushing everything by default would be a security risk for some.

meowface 7y ago
Just to add onto your second point for others who might not be aware, the experimental secret and SSH agent forwarding features have greatly simplified a lot of Dockerfiles I work on. SSH forwarding in particular has been really helpful for dealing with private dependencies. There's a good summary here: https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 https://medium.com/@tonistiigi/build-secrets-and-ssh-forward.... The tl;dr is you can now write lines like "RUN --mount=type=ssh git clone ssh://git@domain/project.git" (or any command that uses SSH) in a Dockerfile to use your host machine's SSH agent during "docker build". You do currently need to specify experimental syntax in the Dockerfile, and set the DOCKER_BUILDKIT environment variable to 1, and pass "--ssh default" to "docker build", but it's a great workflow improvement IMO.

cpuguy83 7y ago

Buildkit has some interesting things around storing build cache information in an image.