3 ms·

The large yum update is a symptom, not the problem: 1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is hug

The large yum update is a symptom, not the problem: 1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is huge. If the upstream maintainer had this base image updated, there would be no updates. Not the developer’s fault. Also, if the developer puts it into production, it’s their responsibility to make sure there are no CVE’s in their code, and hence their responsibility to do a yum update. You touched it, you own it. 2. The Docker tools don’t have a way of flattening the image easily (without doing a save/load). This can be fixed down the road. Telling developers not to do yum updates terrifies me as a security person….

ahoka 11y ago
Maybe the conclusion here is to build your own base images?
- justingood 11y ago
  This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
- fatherlinnux 11y ago
  That solves part of the problem. The problem with base images, are they are the root - I call them Adam and Eve images. They do not solve all of the supply chain problems. Layered images such as Apache and Nginx still may be maintained by operations or even Middlware teams. Each team is responsible for making sure they update their piece of the supply chain. If the operations team (or image supplier) doesn't, the "yum upgrade" (or apt-get) downstream (by the developer or middlware maintainer) will bloat the environment. The problem is, it's a symptom, not the problem. Telling people to NOT do yum updates, just promotes security problems because people upstream aren't keeping up with updates. All of that said, the ONLY sane way to do all of this is with Automation. Hence, the reason for ImageStreams [1], BuildConfigs, DeployConfigs, and Triggers inside of OpenShift. When any piece of the spider web detects a change, all of the dependent layers kick off jobs to rebuild themselves. Also, having good tests is critical. Yeah, most people on the planet haven't thought through this. Yes, Docker images make your life more convenient, at the deploy stage, with the downside that you need to do a LOT more thinking (automation and testing) at build phase. Like shipping containers, we pack them at the factory, but can move them around at will after that. Instead of having dock hands load the ship at the port (which took 45 days in 1906). The supply chain problems are the same either way, but at least Docker gives you a good format to solve them. [1]: https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html https://docs.openshift.com/enterprise/3.0/architecture/core_...
gtrevorjay 11y ago
In no way should the advice be taken to be never `yum update`. Perhaps the article could have been clearer, but the advice is more subtle than it first appears. It ties into point 4 ("Don't use a single layer image"). You need to `yum update` at the right time to make good use of layering and to be able to control updates separately from builds. Imagine you build both Nginx and Apache containers off the same distro image. Rather than having `RUN yum -y update && yum -y clean all` as the first step (which is easy to mess up or forget), you can instead build an "update" layer as a named image with it's own Dockerfile consisting solely of the `yum update`. Now your two application Dockerfiles can simply `from base-update-date`. This means you control when you get updates explicitly orthogonal to the application build and lets you run the two apps off slightly different versions, if there is indeed an issue you need time to resolve regarding backwards compatibility. This also means you get the most out of the caching mechanisms in Docker. Full disclosure - I am both a hatter and a sec guy. [edit, spelling, grammar]
- fatherlinnux 11y ago
  Agreed, that's why I published this: http://rhelblog.redhat.com/2016/02/24/container-tidbits-can-good-supply-chain-hygiene-mitigate-base-image-sizes/ http://rhelblog.redhat.com/2016/02/24/container-tidbits-can-...
- fatherlinnux 11y ago
  Again, this is recursion, if the upstream maintainer for the Apache and Nginx images is doing their job, the "yum update" should do nothing. Though, I agree, it's very important to highlight that too much branching can cause massive environment bloat and way too many permutations. If we just tell everyone to do yum updates at their layer this problem goes away. Ahh, recursion...
  - gtrevorjay 11y ago
    The issue is one of process/collaboration/composition. Even at the largest of distros, package maintainers usually act with pseudo independence. They can change, QA, and push their packages basically at will. Images are collections of those pushes. Even with automation the aggregate nature of images makes them qualitatively different and---especially organizationally---provides a process challenge.
whalesalad 11y ago
Yes to both points. Also, a lot of base images are pretty huge. I am looking forward to the move to alpine-based images (as well as assisting with that process)
- fatherlinnux 11y ago
  Careful, it's not the base image size. It's the environment size. If you don't put enough in your base image, you will have bloat everywhere. If you put too much, it will cause slow start (caching takes a while). It's a balancing act. It's not easy. But, if you use a package exactly twice in your environment, then putting it in the base image actually makes sense. Think at scale, not about individual images.
- fatherlinnux 11y ago
  Also, though I love Alpine, it uses Musl instead of Glibc. Some giant amount of the supply chain uses Glibc. You could convert everything to Musl, but then you would probably have performance hits on all kinds of things, so you would be trading CPU for Disk. Also, this could take TONS of time and money to figure out what is missing in Musl (including syscall mapping (not even sure it exists in other c libraries), etc) Nothing is free, nothing. Building a cool new C library (musl) is fun. Maintaining one for 10, 20, or 30 years (glibc) is work (and fun)...

ahoka 11y ago

Maybe the conclusion here is to build your own base images?

justingood 11y ago
This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
fatherlinnux 11y ago
That solves part of the problem. The problem with base images, are they are the root - I call them Adam and Eve images. They do not solve all of the supply chain problems. Layered images such as Apache and Nginx still may be maintained by operations or even Middlware teams. Each team is responsible for making sure they update their piece of the supply chain. If the operations team (or image supplier) doesn't, the "yum upgrade" (or apt-get) downstream (by the developer or middlware maintainer) will bloat the environment. The problem is, it's a symptom, not the problem. Telling people to NOT do yum updates, just promotes security problems because people upstream aren't keeping up with updates. All of that said, the ONLY sane way to do all of this is with Automation. Hence, the reason for ImageStreams [1], BuildConfigs, DeployConfigs, and Triggers inside of OpenShift. When any piece of the spider web detects a change, all of the dependent layers kick off jobs to rebuild themselves. Also, having good tests is critical. Yeah, most people on the planet haven't thought through this. Yes, Docker images make your life more convenient, at the deploy stage, with the downside that you need to do a LOT more thinking (automation and testing) at build phase. Like shipping containers, we pack them at the factory, but can move them around at will after that. Instead of having dock hands load the ship at the port (which took 45 days in 1906). The supply chain problems are the same either way, but at least Docker gives you a good format to solve them. [1]: https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html https://docs.openshift.com/enterprise/3.0/architecture/core_...

gtrevorjay 11y ago

In no way should the advice be taken to be never `yum update`. Perhaps the article could have been clearer, but the advice is more subtle than it first appears. It ties into point 4 ("Don't use a single layer image"). You need to `yum update` at the right time to make good use of layering and to be able to control updates separately from builds. Imagine you build both Nginx and Apache containers off the same distro image. Rather than having `RUN yum -y update && yum -y clean all` as the first step (which is easy to mess up or forget), you can instead build an "update" layer as a named image with it's own Dockerfile consisting solely of the `yum update`. Now your two application Dockerfiles can simply `from base-update-date`. This means you control when you get updates explicitly orthogonal to the application build and lets you run the two apps off slightly different versions, if there is indeed an issue you need time to resolve regarding backwards compatibility. This also means you get the most out of the caching mechanisms in Docker. Full disclosure - I am both a hatter and a sec guy. [edit, spelling, grammar]

fatherlinnux 11y ago
Agreed, that's why I published this: http://rhelblog.redhat.com/2016/02/24/container-tidbits-can-good-supply-chain-hygiene-mitigate-base-image-sizes/ http://rhelblog.redhat.com/2016/02/24/container-tidbits-can-...
fatherlinnux 11y ago
Again, this is recursion, if the upstream maintainer for the Apache and Nginx images is doing their job, the "yum update" should do nothing. Though, I agree, it's very important to highlight that too much branching can cause massive environment bloat and way too many permutations. If we just tell everyone to do yum updates at their layer this problem goes away. Ahh, recursion...
- gtrevorjay 11y ago
  The issue is one of process/collaboration/composition. Even at the largest of distros, package maintainers usually act with pseudo independence. They can change, QA, and push their packages basically at will. Images are collections of those pushes. Even with automation the aggregate nature of images makes them qualitatively different and---especially organizationally---provides a process challenge.

whalesalad 11y ago

Yes to both points. Also, a lot of base images are pretty huge. I am looking forward to the move to alpine-based images (as well as assisting with that process)

fatherlinnux 11y ago
Careful, it's not the base image size. It's the environment size. If you don't put enough in your base image, you will have bloat everywhere. If you put too much, it will cause slow start (caching takes a while). It's a balancing act. It's not easy. But, if you use a package exactly twice in your environment, then putting it in the base image actually makes sense. Think at scale, not about individual images.
fatherlinnux 11y ago
Also, though I love Alpine, it uses Musl instead of Glibc. Some giant amount of the supply chain uses Glibc. You could convert everything to Musl, but then you would probably have performance hits on all kinds of things, so you would be trading CPU for Disk. Also, this could take TONS of time and money to figure out what is missing in Musl (including syscall mapping (not even sure it exists in other c libraries), etc) Nothing is free, nothing. Building a cool new C library (musl) is fun. Maintaining one for 10, 20, or 30 years (glibc) is work (and fun)...