4 ms·

Before containers everyone used VMs. VMs had a much bigger overhead and were harder to manage. But they were also secure. Now everyone other docker tutorial exp

Before containers everyone used VMs. VMs had a much bigger overhead and were harder to manage. But they were also secure. Now everyone other docker tutorial explains containers as lightweight linux VMs, while in terms of security they are far from it. I don't like the idea of running my (internal, not meant to face the internet) containers so that they are publicly accessible or running their processes as root. Yet this seems to be the default with docker and I'm sure a lot of people don't bother fixing that. There is so much confusion regarding users and user namespaces. I think something needs to change in the way docker documents those things and also in the way defaults are chosen for various configuration options.

ArloL 6y ago
Were they really that significantly more secure? You still needed to do regular maintenance on the underlying image, etc. Same with Docker. The only big difference I see is that yes, breaking out of a container is easier than out of a VM. But are there any other significant vectors I should be aware of?
- plasticxme 6y ago
  They were not more secure, just more isolated. The challenges are different. Containers are just namespaced processes that share the same kernel as the host. A host has access to all container processes, uids, gids, file systems, and networks. Cgroups are used to limit resource access. To run containers securely you need to understand how to protect running processes. You need to use unprivileged users where possible, drop all kernel capabilities not required, run Linux Security Modules (AppArmor, SELinux) to prevent processes from doing things they shouldn’t; and, run containers based on the smallest image possible, since a container should only have files that are absolutely required to run a process, and nothing more. Even when you do it all right, in a multi tenant environment, it’s not safe to run all containers on the same hosts.
  - ArloL 6y ago
    So if I am understanding this correctly the challenges of setting up a secure linux VM and a container are more or less the same? The point about multi-tenancy is absolutely understandable. Isn't this an old story from the PHP world with multi-tenancy? I think a good generalization is: don't run on multi-tenant systems if you do anything (!) critical (e.g. authentication or payments)? But that of course disregards the fact that when people _can_ do something, they _will_ do it even though they shouldn't (like running E-Commerce systems in multi-tenant environments). Another thought regarding isolation: aren't VMs essentially just running on one host as well? Is that why you said "VMs are _more_ isolated"?
- turminal 6y ago
  That's a very important difference, because isolation and the associated increase in overall security of the system is a core purpose of any virtualization technology. Docker promises a lot here, but a lot of those promises remain unfulfilled in reality. Yes, containers are inherently easier to break out of than VMs, but even with that caveat there is room for improvement in container security. That alone is enough reasons for me not to be a big fan of docker in production. But there are other vectors. With a VM you get a whole linux distribution, which of course increases the attack surface, but at the same time you also get much better isolation and that distribution's team of maintainers looking over your software, providing security patches, advisories, a simple way to update the system and so on. On the other hand there exist 'docker best practices' tutorials (not the posted one) that recommend not updating your base system at all in the name of reproducibility. Docker's solution to update management is manual image tagging and manual updates, possibly with help of external tooling. I don't think that's a good solution for that problem. Imo the overall best solution is to run stuff in VMs and pick a lightweight distro for that.
  - ArloL 6y ago
    Just to be sure, isn't a container a whole linux distribution as well depending on your base image? With the same distribution team, etc.? That not updating part is of course just plain and simply bad advice. What solutions for update management would you recommend in the VM space?
imglorp 6y ago
I think it's better to say VMs are _more_ secure. You can never leave out the _more_. Example: https://www.vmware.com/security/advisories.html https://www.vmware.com/security/advisories.html
- cpuguy83 6y ago
  VM's offer different security. That is all. It is more secure to run a process with seccomp filters than it is to run it without. It is more secure to run a process with seccomp and vm isolation than just one or none of these.
MaKey 6y ago
You can use VMs to isolate your containers, Amazon developed Firecracker [0] for that. [0] https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/

ArloL 6y ago

Were they really that significantly more secure? You still needed to do regular maintenance on the underlying image, etc. Same with Docker. The only big difference I see is that yes, breaking out of a container is easier than out of a VM. But are there any other significant vectors I should be aware of?

plasticxme 6y ago
They were not more secure, just more isolated. The challenges are different. Containers are just namespaced processes that share the same kernel as the host. A host has access to all container processes, uids, gids, file systems, and networks. Cgroups are used to limit resource access. To run containers securely you need to understand how to protect running processes. You need to use unprivileged users where possible, drop all kernel capabilities not required, run Linux Security Modules (AppArmor, SELinux) to prevent processes from doing things they shouldn’t; and, run containers based on the smallest image possible, since a container should only have files that are absolutely required to run a process, and nothing more. Even when you do it all right, in a multi tenant environment, it’s not safe to run all containers on the same hosts.
- ArloL 6y ago
  So if I am understanding this correctly the challenges of setting up a secure linux VM and a container are more or less the same? The point about multi-tenancy is absolutely understandable. Isn't this an old story from the PHP world with multi-tenancy? I think a good generalization is: don't run on multi-tenant systems if you do anything (!) critical (e.g. authentication or payments)? But that of course disregards the fact that when people _can_ do something, they _will_ do it even though they shouldn't (like running E-Commerce systems in multi-tenant environments). Another thought regarding isolation: aren't VMs essentially just running on one host as well? Is that why you said "VMs are _more_ isolated"?
turminal 6y ago
That's a very important difference, because isolation and the associated increase in overall security of the system is a core purpose of any virtualization technology. Docker promises a lot here, but a lot of those promises remain unfulfilled in reality. Yes, containers are inherently easier to break out of than VMs, but even with that caveat there is room for improvement in container security. That alone is enough reasons for me not to be a big fan of docker in production. But there are other vectors. With a VM you get a whole linux distribution, which of course increases the attack surface, but at the same time you also get much better isolation and that distribution's team of maintainers looking over your software, providing security patches, advisories, a simple way to update the system and so on. On the other hand there exist 'docker best practices' tutorials (not the posted one) that recommend not updating your base system at all in the name of reproducibility. Docker's solution to update management is manual image tagging and manual updates, possibly with help of external tooling. I don't think that's a good solution for that problem. Imo the overall best solution is to run stuff in VMs and pick a lightweight distro for that.
- ArloL 6y ago
  Just to be sure, isn't a container a whole linux distribution as well depending on your base image? With the same distribution team, etc.? That not updating part is of course just plain and simply bad advice. What solutions for update management would you recommend in the VM space?

imglorp 6y ago

I think it's better to say VMs are _more_ secure. You can never leave out the _more_. Example: https://www.vmware.com/security/advisories.html https://www.vmware.com/security/advisories.html

cpuguy83 6y ago
VM's offer different security. That is all. It is more secure to run a process with seccomp filters than it is to run it without. It is more secure to run a process with seccomp and vm isolation than just one or none of these.

MaKey 6y ago

You can use VMs to isolate your containers, Amazon developed Firecracker [0] for that. [0] https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/