4 ms·

This may be a nice time and place to ask ... I have docker problems. We have a monorepo in GitLab and that works nicely. Many Dockerfiles are present in this r

This may be a nice time and place to ask ... I have docker problems. We have a monorepo in GitLab and that works nicely. Many Dockerfiles are present in this repo in various folders. They all build private docker images pushed to AWS ECR. They all depend on each other and some have multiple parents, so this sort of forms a family tree. It is a major pain to know what docker images are stale and need to be rebuilt, and to ensure that it's parents are also up to date. We do all this manually and it sucks. Some images take hours to build and we only want to rebuild them when necessary. It would be great if the build server cached build-steps so that the resulting images could share more layers with previously downloaded images, saving hugely on disk space, time and bandwidth for people and servers. We make sure to have one Dockerfile per folder and a file called "destination.txt" indicating the final image name adjacent to it, this allows scripts to easily build the entire tree of images and parents by scanning our repo. We want nice automation. I don't even know what the best practices are here. What should I do?

abatilo 8y ago
I would make build triggers that are based on specific Dockerfiles changing in the repo. Many CI systems can watch a specific file being changed as the task trigger. So if container A is a dependency for container B, then make container B's task trigger be whenever the Dockerfile for container A changes.
cutety 8y ago
Several hour builds? I have to ask what kind of app this is for, because I haven’t ever had a build take more than 2 minutes. I’ve been using docker heavily in CI/dev for about 2ish years and the past couple months been getting some production stuff going. My first thought since you say you’re having issues with build caching, you might want to look into multi-stage builds[1], if you haven’t already. I haven’t gotten any “heavy” use from these since I mostly do web stuff in non compiled languages, so I can’t benefit from this feature as much as a compiled language than can just copy binaries around. But, it has been useful for like compiling JS, since it’s done in a separate stage, I a) don’t ever have to contaminate my final image with a node_modules at any point and b) can rebuild the image and used the cached stage if there are no changes in the front end, and that’s usually the longest parts of my builds anyway. However, if you’re in a compiled language (which from the sounds of it, I have a feeling you might be — long builds and multiple dependent components that have to be rebuilt), you can compile some core component in the first stage (A), then have multiple stages (B, C) that depend on that component’s compiled files, and then maybe a final stage (D) that depends on B & C. So then, if you make changes to just C, you’ll only have to rebuild the C & D stages, while the other two are cached. Each stage itself is just an intermediary image, so you can use them as you would the final, and you can even tell Docker to just build up to stage B, if you only need that image. [1] https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...
- etaioinshrdlu 8y ago
  Ahh yes -- some of our dependencies are huge compiled C++ projects that need some customization. We're starting to use multi stage builds here and there. Seems like an essential building block. The main pain point seems to be knowing when an image needs to be rebuilt and to ensure any dependent images also get rebuilt.
temikus 8y ago
Seems like a build pipeline may be the way to go. Check out Concourse CI or Jenkins Pipelines. Those allow to define multi-stage builds with custom triggers, which should allow you to trigger rebuilding dependent images.

abatilo 8y ago

I would make build triggers that are based on specific Dockerfiles changing in the repo. Many CI systems can watch a specific file being changed as the task trigger. So if container A is a dependency for container B, then make container B's task trigger be whenever the Dockerfile for container A changes.

cutety 8y ago

Several hour builds? I have to ask what kind of app this is for, because I haven’t ever had a build take more than 2 minutes. I’ve been using docker heavily in CI/dev for about 2ish years and the past couple months been getting some production stuff going. My first thought since you say you’re having issues with build caching, you might want to look into multi-stage builds[1], if you haven’t already. I haven’t gotten any “heavy” use from these since I mostly do web stuff in non compiled languages, so I can’t benefit from this feature as much as a compiled language than can just copy binaries around. But, it has been useful for like compiling JS, since it’s done in a separate stage, I a) don’t ever have to contaminate my final image with a node_modules at any point and b) can rebuild the image and used the cached stage if there are no changes in the front end, and that’s usually the longest parts of my builds anyway. However, if you’re in a compiled language (which from the sounds of it, I have a feeling you might be — long builds and multiple dependent components that have to be rebuilt), you can compile some core component in the first stage (A), then have multiple stages (B, C) that depend on that component’s compiled files, and then maybe a final stage (D) that depends on B & C. So then, if you make changes to just C, you’ll only have to rebuild the C & D stages, while the other two are cached. Each stage itself is just an intermediary image, so you can use them as you would the final, and you can even tell Docker to just build up to stage B, if you only need that image. [1] https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...

etaioinshrdlu 8y ago
Ahh yes -- some of our dependencies are huge compiled C++ projects that need some customization. We're starting to use multi stage builds here and there. Seems like an essential building block. The main pain point seems to be knowing when an image needs to be rebuilt and to ensure any dependent images also get rebuilt.

temikus 8y ago

Seems like a build pipeline may be the way to go. Check out Concourse CI or Jenkins Pipelines. Those allow to define multi-stage builds with custom triggers, which should allow you to trigger rebuilding dependent images.