3 ms·

The thing I'm always interested in finding out is how people using lots of containers deal with security updates. Do they go into every image and run the updat

The thing I'm always interested in finding out is how people using lots of containers deal with security updates. Do they go into every image and run the updates for the base image it's based on? Does that just mean you're now subject to multiple OS security and patching and best practices? Do they think it doesn't matter and just get new images then they're published, which from what I can see is just when the overplayed software has an update and not the system anduvraries it relies on? When glibc or zlib have a security update, what do you do? For RHEL and derivatives it looks line quay.io tries to help with that, but what's the best practice for the rest of the community? We haven't really adopted containers at all at work yet, and to be honest this is at least part of the reason. It feels like we'd be doing all the patching we currently are, and then a bunch more. Sure, there are some gains, but if also feels like there's a lot more work involved there too?

slekker 3y ago
You can use tools like Snyk to scan images for vulnerabilities (even the Docker(tm) cli tool has one now), you can do things like failing your CI pipeline if there are critica vulnerabilities. You can also use Dependabot (and others) to update your images on a cronjob-like schedule. This is a solved problem
- SkiFire13 3y ago
  This is a solution for the app developer, not for the users that might have to deal with older and unsupported apps.
- Yasuraka 3y ago
  I highly advise going with Trivy over Snyk or, as it's still in beta at the moment, Docker Scout.
  - slekker 3y ago
    Hadn't heard of Trivy, but glad to see another great product from AquaSec :^)
zaptheimpaler 3y ago
So i primarily use containers on my local machine walled off from the internet, so it's not a big concern for me. Watchtower [1] is popular among home server users too which automatically updates containers to the latest image. For production uses I think companies generally build their own containers. They would have a common base linux container and build the other containers based off that with a typical CI/CD pipeline. So if glibc is patched, it's probably patched in the base container and the others are then rebuilt. You don't have to patch each container individually, just the base. Once the whole build pipeline is automated its not too hard to add checks for security updates and rebuild when needed. Production also minimizes the scope of containers with nothing installed except what's necessary so they have few dependencies. [1] https://github.com/containrrr/watchtower https://github.com/containrrr/watchtower
- kbenson 3y ago
  The gist of what I'm reading here and in other comments is that you either have a full build pipeline and rebuild nightly or on demand, and then roll out container updates too, along with OS updates that are needed? If so, that seems like the "a lot more work" I mentioned, because you still have to update all the systems, but now you have a separate process to follow for all the apps/containers on the systems too, and that's assuming you have a build pipeline. Or is the assumption that it's all k8s so the container updates being applied part is automated as well? Because assuming k8s in a discussion about containers does seem to be a bad habit from both detractors and proponents of containers from what I've seen, so I wouldn't be that surprised... (for home user stuff I'm not too worried about finding out if there's a newer image, but I am worried that the image is updated once a year because the publishers of it don't care about base image security)
  - zaptheimpaler 3y ago
    On a single system with `docker compose`, redeploying containers is just `docker compose down && docker compose up`. On clusters you could use k8s or you could use the same orchestration tools you would use to manage a cluster without Docker like Chef/Puppet to run these same commands. Like whatever orchestration system you have for restarting applications without docker, could be used with Docker as well. Personally I do think `k8s` is poorly documented and hard to use so I stay away from it. I don't know if there are good alternatives for production uses though.
chillfox 3y ago
Don’t use glibc… I am of course partially joking. But seriously, use musl libc, build static binaries, build the images from scratch, and have a CI server handle it for you to keep it updated. Alternatively use a small image like Alpine as base if you want some tools in the image for debugging.
otabdeveloper4 3y ago
> ...how people using lots of containers deal with security updates. TL/DR: they don't. #yolo, don't be a square.
rbut 3y ago
We use What's Up Docker [1] to monitor for new versions of docker containers that are created by others (eg. self hosted apps). For containers we create ourselves, we automatically rebuild them each night which pulls the latest security updates. [1] https://github.com/fmartinou/whats-up-docker https://github.com/fmartinou/whats-up-docker
kbenson 3y ago
This is what I get for making a comment on my phone and then watching a movie until the the edit window is closed. s/overplayed/overlaid/ s/anduvraries/and libraries/ s/looks line/looks like/
mynegation 3y ago
As one of those people: we don’t do much, if anything at all. I use containers to try things out behind a VPN like Tailscale. I use containers at work to deploy software but there it is put behind some kind of SSO proxy and lots of different rules apply. Even if I deploy something externally facing, 3rd party containers are going to be on a private docker network - much easier setup than dealing with allow/blocklists and/or firewall rules. Of course I need to take care of my code’s security but that is true with or without containers. For the situations where you deploy 3rd party services facing externally there are solutions (like Watchtower) that I can run and be more confident that this update would not ripple through the system with unknown (and sometimes hidden) consequences.
LorenzoGood 3y ago
I build docker images with Nix, so I just update the library by updating my nix flake.

slekker 3y ago

You can use tools like Snyk to scan images for vulnerabilities (even the Docker(tm) cli tool has one now), you can do things like failing your CI pipeline if there are critica vulnerabilities. You can also use Dependabot (and others) to update your images on a cronjob-like schedule. This is a solved problem

SkiFire13 3y ago
This is a solution for the app developer, not for the users that might have to deal with older and unsupported apps.
Yasuraka 3y ago
I highly advise going with Trivy over Snyk or, as it's still in beta at the moment, Docker Scout.
- slekker 3y ago
  Hadn't heard of Trivy, but glad to see another great product from AquaSec :^)

zaptheimpaler 3y ago

So i primarily use containers on my local machine walled off from the internet, so it's not a big concern for me. Watchtower [1] is popular among home server users too which automatically updates containers to the latest image. For production uses I think companies generally build their own containers. They would have a common base linux container and build the other containers based off that with a typical CI/CD pipeline. So if glibc is patched, it's probably patched in the base container and the others are then rebuilt. You don't have to patch each container individually, just the base. Once the whole build pipeline is automated its not too hard to add checks for security updates and rebuild when needed. Production also minimizes the scope of containers with nothing installed except what's necessary so they have few dependencies. [1] https://github.com/containrrr/watchtower https://github.com/containrrr/watchtower

kbenson 3y ago
The gist of what I'm reading here and in other comments is that you either have a full build pipeline and rebuild nightly or on demand, and then roll out container updates too, along with OS updates that are needed? If so, that seems like the "a lot more work" I mentioned, because you still have to update all the systems, but now you have a separate process to follow for all the apps/containers on the systems too, and that's assuming you have a build pipeline. Or is the assumption that it's all k8s so the container updates being applied part is automated as well? Because assuming k8s in a discussion about containers does seem to be a bad habit from both detractors and proponents of containers from what I've seen, so I wouldn't be that surprised... (for home user stuff I'm not too worried about finding out if there's a newer image, but I am worried that the image is updated once a year because the publishers of it don't care about base image security)
- zaptheimpaler 3y ago
  On a single system with `docker compose`, redeploying containers is just `docker compose down && docker compose up`. On clusters you could use k8s or you could use the same orchestration tools you would use to manage a cluster without Docker like Chef/Puppet to run these same commands. Like whatever orchestration system you have for restarting applications without docker, could be used with Docker as well. Personally I do think `k8s` is poorly documented and hard to use so I stay away from it. I don't know if there are good alternatives for production uses though.

chillfox 3y ago

Don’t use glibc… I am of course partially joking. But seriously, use musl libc, build static binaries, build the images from scratch, and have a CI server handle it for you to keep it updated. Alternatively use a small image like Alpine as base if you want some tools in the image for debugging.

otabdeveloper4 3y ago

> ...how people using lots of containers deal with security updates. TL/DR: they don't. #yolo, don't be a square.

rbut 3y ago

We use What's Up Docker [1] to monitor for new versions of docker containers that are created by others (eg. self hosted apps). For containers we create ourselves, we automatically rebuild them each night which pulls the latest security updates. [1] https://github.com/fmartinou/whats-up-docker https://github.com/fmartinou/whats-up-docker

kbenson 3y ago

This is what I get for making a comment on my phone and then watching a movie until the the edit window is closed. s/overplayed/overlaid/ s/anduvraries/and libraries/ s/looks line/looks like/

mynegation 3y ago

As one of those people: we don’t do much, if anything at all. I use containers to try things out behind a VPN like Tailscale. I use containers at work to deploy software but there it is put behind some kind of SSO proxy and lots of different rules apply. Even if I deploy something externally facing, 3rd party containers are going to be on a private docker network - much easier setup than dealing with allow/blocklists and/or firewall rules. Of course I need to take care of my code’s security but that is true with or without containers. For the situations where you deploy 3rd party services facing externally there are solutions (like Watchtower) that I can run and be more confident that this update would not ripple through the system with unknown (and sometimes hidden) consequences.

LorenzoGood 3y ago

I build docker images with Nix, so I just update the library by updating my nix flake.