4 ms·

The bundling of all your software's dependencies into your container means that you can't upgrade 1 .so on a machine, and all software be updated. Great example

by codemac 10y ago

The bundling of all your software's dependencies into your container means that you can't upgrade 1 .so on a machine, and all software be updated. Great example recently was libgcypt's security bugs. You have to upgrade all containers, which requires figuring out which ones need upgrading. The concern isn't generally that someone will escalate and then delete some other container's filesystem.. it's that once they get remote execution in a container that has connections to your database, http to your discovery service etc, they can steal your customer information. I will say however that this post is showing that by using docker or frankly other well manicured execution enviroments, large swaths of attacks will hopefully not work (selinux & seccomp being great examples).

riskable 10y ago
Well, Docker containers are really meant to be short-lived. On the order of milliseconds (e.g. run a batch operation) to days (run a web server application). They're also meant to be mostly immutable. So a vulnerability in a particular library that allows an attacker access to the container won't necessarily allow them to do much. The other benefit of short-lived containers is that any given vulnerability only lasts as long as the container (assuming you're rebuilding your image with the latest & greatest software/libs before each deployment). Whereas with a full OS/VM you have to find down time to patch that sucker and that might take a month or even a quarter or longer.
- pmontra 10y ago
  I'm not sure I understand this. Even if you keep deploying new versions of containers every few days because you use them to deploy new features, how about the db and web server containers? And how about the vulnerabilities that won't be patched until you make a new deploy? We should build a new container after each apt-get for security patches, right?
  - tayo42 10y ago
    Why are websevers a big deal to redeploy? Why do you want to run a db in a container? That always seemed crazy to me...
    - riskable 10y ago
      Running a traditional relational DB inside a container is pretty stupid. It can take a while for those suckers to stop & start and they often don't work properly if placed behind a load balancer with random distribution of connections. Also, if a connection gets cut off in the middle of a transactions things can go very, very wrong. On the other hand, if you architect your database usage in such a way as to ensure that cutoff transactions get detected and retried without relying on persistent connections it can be a good idea to use containers for hosting your database(s). Of course, your use case needs to fit this kind of model for it to work. If your use case requires mostly serial transactions and can't work with "eventual consistency" then containers are probably a no-go.
  - riskable 10y ago
    > We should build a new container after each apt-get for security patches, right? Yes. That's the idea. Here's when and why you re-deploy: * A new, updated version of software/a package is out. * You've made changes to your app. Even something as simple as a typo. * Your container has been up too long (e.g. 24 hours). Replacing existing containers multiple times a day in a production environment is the norm. "The only constant is change" and the more often things change the more often you'll be swapping out your containers for new ones. The whole point of containers is that they can be brought up and down in an instant without impact (if you architect things correctly). So even a VM that runs "apt-get update ; apt-get -y upgrade" every night may not be as up-to-date or secure as the same software running inside a container.
fapjacks 10y ago
If you're using an image tagged "latest" (e.g. FROM ubuntu:latest), updates are already being pulled into your containers by default. Since containers are ephemeral, fixing security bugs can be as simple as just rebuilding and redeploying your containers.
ralmeida 10y ago
That's a good point, for sure. But on the other hand, some upgrades may be delayed or left undone because more services depend on the same library, so the risk of breakage is higher. So, there's at least some incentive to using more recent versions, because there's less surface area to test. Not sure how much this affects what's in the wild, though - I mean, I have no data to show that people really do tend to upgrade more often while containerized than not.
bigmac 10y ago
The important metric with patching vulnerabilities is time-to-patch. Docker based environments are able to significantly reduce time-to-patch precisely because the libs are bundled with the application. Most orgs have trouble rolling out patches to system libs because the testing matrix for rolling out patches mandates testing across the board for all application and system-level consumers of the lib. This can often take weeks or months. When the lib travels bundled with the app, just the app in isolation can be patched and rolled out immediately. It is important to have an inventory so you can do patching across the board and have notifications that it's needed. This is why there are now a number of security scanners for Docker containers, including Docker's own: https://blog.docker.com/2016/05/docker-security-scanning/ https://blog.docker.com/2016/05/docker-security-scanning/ Disclaimer: I manage security at Docker.
bandrami 10y ago
So it's the static/dynamic linking argument all over again. Personally, as a sysadmin, I dislike docker. Not that there's anything wrong with its implementation in particular, but because I can no longer confidently say what is actually in our stack (we run 4 different versions of Python, last I checked -- it's absolute madness). Now, I grant Docker's argument: if you're going to do something like that, Docker is pretty much the best way around to do it. Fair enough: it's the best available tourniquet for the self-inflicted wound of bad stack management. But to me, the ease of redeploying gets outweighed by the increased complexity of the security reasoning you have to do.

by codemac 10y ago

riskable 10y ago

Well, Docker containers are really meant to be short-lived. On the order of milliseconds (e.g. run a batch operation) to days (run a web server application). They're also meant to be mostly immutable. So a vulnerability in a particular library that allows an attacker access to the container won't necessarily allow them to do much. The other benefit of short-lived containers is that any given vulnerability only lasts as long as the container (assuming you're rebuilding your image with the latest & greatest software/libs before each deployment). Whereas with a full OS/VM you have to find down time to patch that sucker and that might take a month or even a quarter or longer.

pmontra 10y ago
I'm not sure I understand this. Even if you keep deploying new versions of containers every few days because you use them to deploy new features, how about the db and web server containers? And how about the vulnerabilities that won't be patched until you make a new deploy? We should build a new container after each apt-get for security patches, right?
- tayo42 10y ago
  Why are websevers a big deal to redeploy? Why do you want to run a db in a container? That always seemed crazy to me...
  - riskable 10y ago
    Running a traditional relational DB inside a container is pretty stupid. It can take a while for those suckers to stop & start and they often don't work properly if placed behind a load balancer with random distribution of connections. Also, if a connection gets cut off in the middle of a transactions things can go very, very wrong. On the other hand, if you architect your database usage in such a way as to ensure that cutoff transactions get detected and retried without relying on persistent connections it can be a good idea to use containers for hosting your database(s). Of course, your use case needs to fit this kind of model for it to work. If your use case requires mostly serial transactions and can't work with "eventual consistency" then containers are probably a no-go.
- riskable 10y ago
  > We should build a new container after each apt-get for security patches, right? Yes. That's the idea. Here's when and why you re-deploy: * A new, updated version of software/a package is out. * You've made changes to your app. Even something as simple as a typo. * Your container has been up too long (e.g. 24 hours). Replacing existing containers multiple times a day in a production environment is the norm. "The only constant is change" and the more often things change the more often you'll be swapping out your containers for new ones. The whole point of containers is that they can be brought up and down in an instant without impact (if you architect things correctly). So even a VM that runs "apt-get update ; apt-get -y upgrade" every night may not be as up-to-date or secure as the same software running inside a container.

fapjacks 10y ago

If you're using an image tagged "latest" (e.g. FROM ubuntu:latest), updates are already being pulled into your containers by default. Since containers are ephemeral, fixing security bugs can be as simple as just rebuilding and redeploying your containers.

ralmeida 10y ago

That's a good point, for sure. But on the other hand, some upgrades may be delayed or left undone because more services depend on the same library, so the risk of breakage is higher. So, there's at least some incentive to using more recent versions, because there's less surface area to test. Not sure how much this affects what's in the wild, though - I mean, I have no data to show that people really do tend to upgrade more often while containerized than not.

bigmac 10y ago

The important metric with patching vulnerabilities is time-to-patch. Docker based environments are able to significantly reduce time-to-patch precisely because the libs are bundled with the application. Most orgs have trouble rolling out patches to system libs because the testing matrix for rolling out patches mandates testing across the board for all application and system-level consumers of the lib. This can often take weeks or months. When the lib travels bundled with the app, just the app in isolation can be patched and rolled out immediately. It is important to have an inventory so you can do patching across the board and have notifications that it's needed. This is why there are now a number of security scanners for Docker containers, including Docker's own: https://blog.docker.com/2016/05/docker-security-scanning/ https://blog.docker.com/2016/05/docker-security-scanning/ Disclaimer: I manage security at Docker.

bandrami 10y ago

So it's the static/dynamic linking argument all over again. Personally, as a sysadmin, I dislike docker. Not that there's anything wrong with its implementation in particular, but because I can no longer confidently say what is actually in our stack (we run 4 different versions of Python, last I checked -- it's absolute madness). Now, I grant Docker's argument: if you're going to do something like that, Docker is pretty much the best way around to do it. Fair enough: it's the best available tourniquet for the self-inflicted wound of bad stack management. But to me, the ease of redeploying gets outweighed by the increased complexity of the security reasoning you have to do.