9 ms·

I find it somewhat curious, that the article starts with listing examples where root user is used to run containers (amongst other issues): > Writing productio

I find it somewhat curious, that the article starts with listing examples where root user is used to run containers (amongst other issues): > Writing production-worthy Dockerfiles is, unfortunately, not as simple as you would imagine. Most Docker images in the wild fail here, and even professionals often[1] get[2] this[2] wrong[3]. And yet, in some of the linked URLs, people are presenting reasons for why that approach was used in particular, instead of the supposedly safer alternatives. For example, https://github.com/caddyserver/caddy-docker/issues/104 https://github.com/caddyserver/caddy-docker/issues/104 > We actually originally did run as non-root by default, but simplicity we decided to drop that (see #24, and also #103 for some other related discussion). > If your Dockerfile works for you, that's great. In most cases where users want to run as non-root, they also don't need to listen to :80/:443 in the container, so the setpcap magic isn't necessary at all. > It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either. > Finally, it's worth considering why you want to run as non-root. What attack vectors are you trying to avoid? Container escape vulnerabilities are pretty much the only real risk, but anyone running a modern Docker version is immune to many of them. It's also worth considering user namespace remapping as a mitigation. In my experience the main reason for running as non-root is to pass compliance checks - not a bad reason, but it's also worth recognizing that non-compliance does not automatically equal decreased security (and vice-versa). If larger projects, like Nginx, Traefik, Httpd and HAProxy were all creating containers like that, it makes you think about the reasoning behind it. Is it easier to just run containers as root and not worry about the permissions inside of the container? If so, wouldn't it really make more sense for the container runtime to have some sort of mechanisms in place to allow people to do what's easy within the containers while also making sure that it has no harmful impact outside of them? Because to me it seems like people will continuously take the path of least resistance and from where i stand, it should be up to the creators of the container technologies to make sure that this path is safe by default.

rualca 6y ago
> It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either. That does not sound like an adequate excuse, does it? I'm afraid none of your remarks makes a reasonable point. Even if you believe that running random stuff as root is ok, if you do not have any reason to do that then why should we mindlessly follow bad practices? If we aim at running a safe system.and there is absolutely zero drawback in following best practices, then why should we continue to make the mistake of intentionally using poor practices?
- LucidLynx 6y ago
  It is definitely possible to run nginx as non-root in a container: https://www.rockyourcode.com/run-docker-nginx-as-non-root-user/ https://www.rockyourcode.com/run-docker-nginx-as-non-root-us... Also, all the sentences that begin with '>' were comments from https://github.com/caddyserver/caddy-docker/issues/104 https://github.com/caddyserver/caddy-docker/issues/104, and not from the user gbrindisi
  - francislavoie 6y ago
    It's also possible to run Caddy in Docker as non-root, but it requires jumping through some hoops when configuring Caddy, for example changing the HTTP and HTTPS ports in global configuration to something else, since those ports are necessary to use for ACME. If the official Docker image did that, then most users would be very confused and we would get lots of support complaints. A cost-benefit analysis told us it was not worth the headache to run as non-root since the Caddy project values highly user experience.
    - cpuguy83 6y ago
      I believe we setup containers in Docker 20.10 such that containers can bind to < 1024 by default without giving it cap_net_bind: https://github.com/moby/moby/pull/41030 https://github.com/moby/moby/pull/41030
- 6y ago
zeroxfe 6y ago
> Is it easier to just run containers as root and not worry about the permissions inside of the container? If so, wouldn't it really make more sense for the container runtime to have some sort of mechanisms in place to allow people to do what's easy within the containers while also making sure that it has no harmful impact outside of them? This is correct in principle, but very hard in practice. This is because kernel support for containers were kind of "tacked on" and more-or-less scattered across the code-base. And although they're getting a lot better, there's still no easy way to reason about their security. So a lot of the advice around permissions management and access control are a kind of defence-in-depth. I'd love it if we get to a point where containers can make strong statements around security.
turminal 6y ago
Before containers everyone used VMs. VMs had a much bigger overhead and were harder to manage. But they were also secure. Now everyone other docker tutorial explains containers as lightweight linux VMs, while in terms of security they are far from it. I don't like the idea of running my (internal, not meant to face the internet) containers so that they are publicly accessible or running their processes as root. Yet this seems to be the default with docker and I'm sure a lot of people don't bother fixing that. There is so much confusion regarding users and user namespaces. I think something needs to change in the way docker documents those things and also in the way defaults are chosen for various configuration options.
- ArloL 6y ago
  Were they really that significantly more secure? You still needed to do regular maintenance on the underlying image, etc. Same with Docker. The only big difference I see is that yes, breaking out of a container is easier than out of a VM. But are there any other significant vectors I should be aware of?
  - plasticxme 6y ago
    They were not more secure, just more isolated. The challenges are different. Containers are just namespaced processes that share the same kernel as the host. A host has access to all container processes, uids, gids, file systems, and networks. Cgroups are used to limit resource access. To run containers securely you need to understand how to protect running processes. You need to use unprivileged users where possible, drop all kernel capabilities not required, run Linux Security Modules (AppArmor, SELinux) to prevent processes from doing things they shouldn’t; and, run containers based on the smallest image possible, since a container should only have files that are absolutely required to run a process, and nothing more. Even when you do it all right, in a multi tenant environment, it’s not safe to run all containers on the same hosts.
    - ArloL 6y ago
      So if I am understanding this correctly the challenges of setting up a secure linux VM and a container are more or less the same? The point about multi-tenancy is absolutely understandable. Isn't this an old story from the PHP world with multi-tenancy? I think a good generalization is: don't run on multi-tenant systems if you do anything (!) critical (e.g. authentication or payments)? But that of course disregards the fact that when people _can_ do something, they _will_ do it even though they shouldn't (like running E-Commerce systems in multi-tenant environments). Another thought regarding isolation: aren't VMs essentially just running on one host as well? Is that why you said "VMs are _more_ isolated"?
plasticxme 6y ago
Nginx and the like are starting to provide non-privileged versions of their container images. Running as root is lazy and equals container escape, especially when running on anything other than scratch and read only file system. The only reason Nginx and Traefik run as root is to bind to privileged ports (80,443). There is no reason to do that inside of a container, since you can remap exposed ports outside of the container. Containers are not VMs and must be handled differently. You are always one RCE away from having your entire container platform compromised.
- ofrzeta 6y ago
  You don't need root to open ports 80 and 443 but instead use CAP_NET_BIND_SERVICE that you can also grant to the container.
  - plasticxme 6y ago
    CAP_NET_BIND_SERVICE is a root privilege, a distinct one provided by kernel capabilities, granted to a process. In order to use it the container must be permitted to allow its processes to elevate their privileges. If the container is running as root permitting it is redundant, since the kernel doesn’t filter root for kernel capabilities anyways. If a privileged user sets CAP_NET_BIND_SERVICE on an executable binary using setpcap to allow a non-root user the ability in a container to bind to a privileged port, elevated privileges are still required for execve to create a process that is permitted to use the kernel capability. Think sudo but for processes. The argument with containers is that binding to a privileged port isn’t necessary, so you shouldn’t do it. And by not doing it you improve your security posture.
  - cpuguy83 6y ago
    Don't even need that on newer kernels and Docker 20.10: https://github.com/moby/moby/pull/41030 https://github.com/moby/moby/pull/41030
- deleted 6y ago
  [deleted]

rualca 6y ago

> It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either. That does not sound like an adequate excuse, does it? I'm afraid none of your remarks makes a reasonable point. Even if you believe that running random stuff as root is ok, if you do not have any reason to do that then why should we mindlessly follow bad practices? If we aim at running a safe system.and there is absolutely zero drawback in following best practices, then why should we continue to make the mistake of intentionally using poor practices?

LucidLynx 6y ago
It is definitely possible to run nginx as non-root in a container: https://www.rockyourcode.com/run-docker-nginx-as-non-root-user/ https://www.rockyourcode.com/run-docker-nginx-as-non-root-us... Also, all the sentences that begin with '>' were comments from https://github.com/caddyserver/caddy-docker/issues/104 https://github.com/caddyserver/caddy-docker/issues/104, and not from the user gbrindisi
- francislavoie 6y ago
  It's also possible to run Caddy in Docker as non-root, but it requires jumping through some hoops when configuring Caddy, for example changing the HTTP and HTTPS ports in global configuration to something else, since those ports are necessary to use for ACME. If the official Docker image did that, then most users would be very confused and we would get lots of support complaints. A cost-benefit analysis told us it was not worth the headache to run as non-root since the Caddy project values highly user experience.
  - cpuguy83 6y ago
    I believe we setup containers in Docker 20.10 such that containers can bind to < 1024 by default without giving it cap_net_bind: https://github.com/moby/moby/pull/41030 https://github.com/moby/moby/pull/41030
6y ago

zeroxfe 6y ago

> Is it easier to just run containers as root and not worry about the permissions inside of the container? If so, wouldn't it really make more sense for the container runtime to have some sort of mechanisms in place to allow people to do what's easy within the containers while also making sure that it has no harmful impact outside of them? This is correct in principle, but very hard in practice. This is because kernel support for containers were kind of "tacked on" and more-or-less scattered across the code-base. And although they're getting a lot better, there's still no easy way to reason about their security. So a lot of the advice around permissions management and access control are a kind of defence-in-depth. I'd love it if we get to a point where containers can make strong statements around security.

turminal 6y ago

Before containers everyone used VMs. VMs had a much bigger overhead and were harder to manage. But they were also secure. Now everyone other docker tutorial explains containers as lightweight linux VMs, while in terms of security they are far from it. I don't like the idea of running my (internal, not meant to face the internet) containers so that they are publicly accessible or running their processes as root. Yet this seems to be the default with docker and I'm sure a lot of people don't bother fixing that. There is so much confusion regarding users and user namespaces. I think something needs to change in the way docker documents those things and also in the way defaults are chosen for various configuration options.

ArloL 6y ago
Were they really that significantly more secure? You still needed to do regular maintenance on the underlying image, etc. Same with Docker. The only big difference I see is that yes, breaking out of a container is easier than out of a VM. But are there any other significant vectors I should be aware of?
- plasticxme 6y ago
  They were not more secure, just more isolated. The challenges are different. Containers are just namespaced processes that share the same kernel as the host. A host has access to all container processes, uids, gids, file systems, and networks. Cgroups are used to limit resource access. To run containers securely you need to understand how to protect running processes. You need to use unprivileged users where possible, drop all kernel capabilities not required, run Linux Security Modules (AppArmor, SELinux) to prevent processes from doing things they shouldn’t; and, run containers based on the smallest image possible, since a container should only have files that are absolutely required to run a process, and nothing more. Even when you do it all right, in a multi tenant environment, it’s not safe to run all containers on the same hosts.
  - ArloL 6y ago
    So if I am understanding this correctly the challenges of setting up a secure linux VM and a container are more or less the same? The point about multi-tenancy is absolutely understandable. Isn't this an old story from the PHP world with multi-tenancy? I think a good generalization is: don't run on multi-tenant systems if you do anything (!) critical (e.g. authentication or payments)? But that of course disregards the fact that when people _can_ do something, they _will_ do it even though they shouldn't (like running E-Commerce systems in multi-tenant environments). Another thought regarding isolation: aren't VMs essentially just running on one host as well? Is that why you said "VMs are _more_ isolated"?

plasticxme 6y ago

Nginx and the like are starting to provide non-privileged versions of their container images. Running as root is lazy and equals container escape, especially when running on anything other than scratch and read only file system. The only reason Nginx and Traefik run as root is to bind to privileged ports (80,443). There is no reason to do that inside of a container, since you can remap exposed ports outside of the container. Containers are not VMs and must be handled differently. You are always one RCE away from having your entire container platform compromised.

ofrzeta 6y ago
You don't need root to open ports 80 and 443 but instead use CAP_NET_BIND_SERVICE that you can also grant to the container.
- plasticxme 6y ago
  CAP_NET_BIND_SERVICE is a root privilege, a distinct one provided by kernel capabilities, granted to a process. In order to use it the container must be permitted to allow its processes to elevate their privileges. If the container is running as root permitting it is redundant, since the kernel doesn’t filter root for kernel capabilities anyways. If a privileged user sets CAP_NET_BIND_SERVICE on an executable binary using setpcap to allow a non-root user the ability in a container to bind to a privileged port, elevated privileges are still required for execve to create a process that is permitted to use the kernel capability. Think sudo but for processes. The argument with containers is that binding to a privileged port isn’t necessary, so you shouldn’t do it. And by not doing it you improve your security posture.
- cpuguy83 6y ago
  Don't even need that on newer kernels and Docker 20.10: https://github.com/moby/moby/pull/41030 https://github.com/moby/moby/pull/41030
deleted 6y ago
[deleted]