5 ms·

Could someone comment on how secure such a container is, at least nominally? Should I be able to theoretically run untrusted code on such a container if the sys

Could someone comment on how secure such a container is, at least nominally? Should I be able to theoretically run untrusted code on such a container if the system is bug-free and I add proper error-checking to the code? Or are there things that you'd need to worry about the code being able to access? Any considerations regarding sudo permissions?

lmuench 6y ago
Docker has a concept of layered images where only the top-layer is writable. The layer above the "scratch"[1] image usually contain all the files of the base images OS and that's what you set your root directory to. The writable layer disappears when the container is stopped. If you mount your DIY container into /tmp for example, the process running inside your container won't be able to access any OS functionality. You couldn't run a web server in such a container for instance. On the other hand, whatever your containerized process writes into the mounted part of your hard disk won't disappear when the container stops. Because of that, I wouldn't run untrusted code in it. [1] https://hub.docker.com/_/scratch https://hub.docker.com/_/scratch
- rantwasp 6y ago
  this is the file system not docker itself. you can get the overlay behavior without any docker
  - lmuench 6y ago
    Can you point me to some online resources? I'd like to learn more about this.
    - mehrdadn 6y ago
      I'm not sure if this is what the parent is referring to but there are overlayfs and unionfs in Ubuntu for example.
    - rantwasp 6y ago
      start with: https://windsock.io/the-overlay-filesystem/ https://windsock.io/the-overlay-filesystem/ after that read more about overlay, overlay fs, for historical reasons aufs.
    - throwaway8941 6y ago
      https://lwn.net/Articles/324291/ https://lwn.net/Articles/324291/ https://lwn.net/Articles/325369/ https://lwn.net/Articles/325369/ https://lwn.net/Articles/327738/ https://lwn.net/Articles/327738/
      - rantwasp 6y ago
        nice. thank you
- mehrdadn 6y ago
  I see, thanks. It's just a matter of preparing the directory before and cleaning it up afterward though, right? Not a security hole exactly?
  - lmuench 6y ago
    Correct.
rantwasp 6y ago
here is a secret: there is no such thing as a container. it’s an abstraction we made up and containers rely on kernel features. if you use those features correctly it’s as secure as it gets - chances are that if you’re going to roll your own you’ll miss some things.
- mehrdadn 6y ago
  That doesn't answer my question at all. I'm well aware containers aren't a "real thing". My entire question was about the "if" part that you didn't address. Is anything missed here? is my question.
  - rantwasp 6y ago
    in theory, it’s a secure as the kernel if your code does the right thing. that being said, the attack surface is wider than say if you would run it in a VM or its own physical machine
    - mehrdadn 6y ago
      Well obviously "if your code does the right thing" then it's going to be secure rather than insecure... by definition. That's again a pretty unhelpful tautology. I'm asking about the code in the webpage, not code I'm writing personally. I'm saying let's assume it has error-handling added to it. That's it. I am not writing any code otherwise. Is that code doing "the right thing"? Or are there more things it needs to be doing?
zajio1am 6y ago
The code from the article misses an important thing w.r.t. security - user namespaces, which can be used to map container UIDs to a subset of host UIDs and container root to non-root user. But it is a namespace that is more complicated to configure than others.
- mehrdadn 6y ago
  Ah thank you! That's what I was looking for.
jeffbee 6y ago
Big “if”. There has never, in thirty years, been a Linux that lacked a user-to-root privilege escalation path. Running untrusted code in containers is the same as it’s ever been: totally unsafe. VMs are safer, or, minimally, ptrace sandboxes intercepting all syscalls.
- mehrdadn 6y ago
  Are you saying because of bugs or are you saying it's by design? I explicitly said ignore OS bugs.
  - jeffbee 6y ago
    Yes you did, but that’s as useless as a discussion based on ignoring the laws of thermodynamics.
    - mehrdadn 6y ago
      If you find it useless that doesn't mean I do. There's no obligation to contribute if you don't have anything to.
  - effie 6y ago
    That may be hard. Some bugs are elevated to features and then become part of the design.
    - mehrdadn 6y ago
      This is too vague to be useful.
- deathanatos 6y ago
  One can write the same sentence about VMs. The most recent Xen escalation bug I can find in 5 minutes of Googling has a publication date of Jan 2020. Emulating an entire machine, all of its diverse hardware, their bespoke protocols, and all the weirdness of the x86/amd64 ISA is fraught with peril. It is a large attack space. So too is the Linux kernel. And frankly, inside the VM, half of us are running Linux anyways. I feel like for a lot of use cases, compromising the VM's guest OS (Linux) is enough to have a really bad day. Compromising the hypervisor? Bad, yes, but now it's AWS's problem. There's more to containers, too, than just the security thing, and I think there are enough other advantages (I can more easily bin-pack services together; I can separate the FS and thus dependencies of unrelated components; I can more easily test them; etc.) that containers are worth it. Often and even on top of a VM. (My current work is with containers, and we run them on VMs.)
vocram 6y ago
The container process has still full access to /dev, /sys, all capabilities of root, and the ability to insmod.
- mehrdadn 6y ago
  I didn't notice that, thanks! Although I imagine they can't do much with /dev etc. unless they get sudo.
  - cyphar 6y ago
    Unless you're using user namespaces (which this doesn't) then root inside a container is equal to root outside the container. You don't even need access to /dev, because the container process could just mknod(2) any device and access it with full permissions. This is only possible in this example because the container has the full capability set (including CAP_MKNOD) and the devices cgroup hasn't been configured to restrict device access. Real container runtimes always restrict device creation by default, and usually don't allow CAP_MKNOD by default.
    - mehrdadn 6y ago
      Thanks!
uvuv 6y ago
Definitely not secure. The author did a great job explaining container runtimes in basic terms, but there's a lot of security features missing. Mainly: * Reducing the container's capabilities * Restricting access to resources through cgroups * Applying seccomp filters to prevent certain syscalls. As another comment suggested, user namespaces are another hardening feature, but not all container runtimes enable it by default. Podman does, Docker doesn't. In fact user namespaces are so powerful that I believe they pretty much cover most of the hardening provided by the three features I listed above. If you're wondering why they're not enabled by default in Docker, take a look at this [1]. Exploiting the missing isolation mechanisms, the following bash commands will allow you to escape from the author's containers: $ ls -al /sys/dev/block # find the root fs device (e.g. /dev/sda1) major and minor device numbers (e.g. {maj=8, min=1}, {maj=259, min=1}) $ mknod --mode 0600 /dev/host_fs_dev b $major $minor $ mkdir /host_fs && mount /dev/host_fs_dev /host_fs (warning: shameless plug to my posts follows:) If you want more details, I wrote a post on this exact same problem in the context of three vulnerabilities I found in rkt (another container runtime) [2]. Beside the issues above, the author's runtime also exposes host file descriptors like /proc/self/exe that can be used to escape the container. This is a post I wrote on runC CVE-2019-5736 that explains this kind of issues. [1] https://docs.docker.com/engine/security/userns-remap/#user-namespace-known-limitations https://docs.docker.com/engine/security/userns-remap/#user-n... [2] https://unit42.paloaltonetworks.com/breaking-out-of-coresos-rkt-3-new-cves/ https://unit42.paloaltonetworks.com/breaking-out-of-coresos-... [3] https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-cve-2019-5736/ https://unit42.paloaltonetworks.com/breaking-docker-via-runc...
- mehrdadn 6y ago
  Wow thank you!!
- rdslw 6y ago
  Thank you for detailed answer and interesting links! Could you please explain/point me to some information/source, why docker can't use -net=host namespace if userns is enabled, while on the other hand rootlesskit[1] which uses userns by default, dont have problem with using host netns (--net=host) ? [1] https://github.com/rootless-containers/rootlesskit https://github.com/rootless-containers/rootlesskit

lmuench 6y ago

Docker has a concept of layered images where only the top-layer is writable. The layer above the "scratch"[1] image usually contain all the files of the base images OS and that's what you set your root directory to. The writable layer disappears when the container is stopped. If you mount your DIY container into /tmp for example, the process running inside your container won't be able to access any OS functionality. You couldn't run a web server in such a container for instance. On the other hand, whatever your containerized process writes into the mounted part of your hard disk won't disappear when the container stops. Because of that, I wouldn't run untrusted code in it. [1] https://hub.docker.com/_/scratch https://hub.docker.com/_/scratch

rantwasp 6y ago
this is the file system not docker itself. you can get the overlay behavior without any docker
- lmuench 6y ago
  Can you point me to some online resources? I'd like to learn more about this.
  - mehrdadn 6y ago
    I'm not sure if this is what the parent is referring to but there are overlayfs and unionfs in Ubuntu for example.
  - rantwasp 6y ago
    start with: https://windsock.io/the-overlay-filesystem/ https://windsock.io/the-overlay-filesystem/ after that read more about overlay, overlay fs, for historical reasons aufs.
  - throwaway8941 6y ago
    https://lwn.net/Articles/324291/ https://lwn.net/Articles/324291/ https://lwn.net/Articles/325369/ https://lwn.net/Articles/325369/ https://lwn.net/Articles/327738/ https://lwn.net/Articles/327738/
    - rantwasp 6y ago
      nice. thank you
mehrdadn 6y ago
I see, thanks. It's just a matter of preparing the directory before and cleaning it up afterward though, right? Not a security hole exactly?
- lmuench 6y ago
  Correct.

rantwasp 6y ago

here is a secret: there is no such thing as a container. it’s an abstraction we made up and containers rely on kernel features. if you use those features correctly it’s as secure as it gets - chances are that if you’re going to roll your own you’ll miss some things.

mehrdadn 6y ago
That doesn't answer my question at all. I'm well aware containers aren't a "real thing". My entire question was about the "if" part that you didn't address. Is anything missed here? is my question.
- rantwasp 6y ago
  in theory, it’s a secure as the kernel if your code does the right thing. that being said, the attack surface is wider than say if you would run it in a VM or its own physical machine
  - mehrdadn 6y ago
    Well obviously "if your code does the right thing" then it's going to be secure rather than insecure... by definition. That's again a pretty unhelpful tautology. I'm asking about the code in the webpage, not code I'm writing personally. I'm saying let's assume it has error-handling added to it. That's it. I am not writing any code otherwise. Is that code doing "the right thing"? Or are there more things it needs to be doing?

zajio1am 6y ago

The code from the article misses an important thing w.r.t. security - user namespaces, which can be used to map container UIDs to a subset of host UIDs and container root to non-root user. But it is a namespace that is more complicated to configure than others.

mehrdadn 6y ago
Ah thank you! That's what I was looking for.

jeffbee 6y ago

Big “if”. There has never, in thirty years, been a Linux that lacked a user-to-root privilege escalation path. Running untrusted code in containers is the same as it’s ever been: totally unsafe. VMs are safer, or, minimally, ptrace sandboxes intercepting all syscalls.

mehrdadn 6y ago
Are you saying because of bugs or are you saying it's by design? I explicitly said ignore OS bugs.
- jeffbee 6y ago
  Yes you did, but that’s as useless as a discussion based on ignoring the laws of thermodynamics.
  - mehrdadn 6y ago
    If you find it useless that doesn't mean I do. There's no obligation to contribute if you don't have anything to.
- effie 6y ago
  That may be hard. Some bugs are elevated to features and then become part of the design.
  - mehrdadn 6y ago
    This is too vague to be useful.
deathanatos 6y ago
One can write the same sentence about VMs. The most recent Xen escalation bug I can find in 5 minutes of Googling has a publication date of Jan 2020. Emulating an entire machine, all of its diverse hardware, their bespoke protocols, and all the weirdness of the x86/amd64 ISA is fraught with peril. It is a large attack space. So too is the Linux kernel. And frankly, inside the VM, half of us are running Linux anyways. I feel like for a lot of use cases, compromising the VM's guest OS (Linux) is enough to have a really bad day. Compromising the hypervisor? Bad, yes, but now it's AWS's problem. There's more to containers, too, than just the security thing, and I think there are enough other advantages (I can more easily bin-pack services together; I can separate the FS and thus dependencies of unrelated components; I can more easily test them; etc.) that containers are worth it. Often and even on top of a VM. (My current work is with containers, and we run them on VMs.)

vocram 6y ago

The container process has still full access to /dev, /sys, all capabilities of root, and the ability to insmod.

mehrdadn 6y ago
I didn't notice that, thanks! Although I imagine they can't do much with /dev etc. unless they get sudo.
- cyphar 6y ago
  Unless you're using user namespaces (which this doesn't) then root inside a container is equal to root outside the container. You don't even need access to /dev, because the container process could just mknod(2) any device and access it with full permissions. This is only possible in this example because the container has the full capability set (including CAP_MKNOD) and the devices cgroup hasn't been configured to restrict device access. Real container runtimes always restrict device creation by default, and usually don't allow CAP_MKNOD by default.
  - mehrdadn 6y ago
    Thanks!

uvuv 6y ago

Definitely not secure. The author did a great job explaining container runtimes in basic terms, but there's a lot of security features missing. Mainly: * Reducing the container's capabilities * Restricting access to resources through cgroups * Applying seccomp filters to prevent certain syscalls. As another comment suggested, user namespaces are another hardening feature, but not all container runtimes enable it by default. Podman does, Docker doesn't. In fact user namespaces are so powerful that I believe they pretty much cover most of the hardening provided by the three features I listed above. If you're wondering why they're not enabled by default in Docker, take a look at this [1]. Exploiting the missing isolation mechanisms, the following bash commands will allow you to escape from the author's containers: $ ls -al /sys/dev/block # find the root fs device (e.g. /dev/sda1) major and minor device numbers (e.g. {maj=8, min=1}, {maj=259, min=1}) $ mknod --mode 0600 /dev/host_fs_dev b $major $minor $ mkdir /host_fs && mount /dev/host_fs_dev /host_fs (warning: shameless plug to my posts follows:) If you want more details, I wrote a post on this exact same problem in the context of three vulnerabilities I found in rkt (another container runtime) [2]. Beside the issues above, the author's runtime also exposes host file descriptors like /proc/self/exe that can be used to escape the container. This is a post I wrote on runC CVE-2019-5736 that explains this kind of issues. [1] https://docs.docker.com/engine/security/userns-remap/#user-namespace-known-limitations https://docs.docker.com/engine/security/userns-remap/#user-n... [2] https://unit42.paloaltonetworks.com/breaking-out-of-coresos-rkt-3-new-cves/ https://unit42.paloaltonetworks.com/breaking-out-of-coresos-... [3] https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-cve-2019-5736/ https://unit42.paloaltonetworks.com/breaking-docker-via-runc...

mehrdadn 6y ago
Wow thank you!!
rdslw 6y ago
Thank you for detailed answer and interesting links! Could you please explain/point me to some information/source, why docker can't use -net=host namespace if userns is enabled, while on the other hand rootlesskit[1] which uses userns by default, dont have problem with using host netns (--net=host) ? [1] https://github.com/rootless-containers/rootlesskit https://github.com/rootless-containers/rootlesskit