5 ms·

This is great! I'd like to see a formal container security grade that works like: 1) Curate a list of all known (container) exploits 2) Run each exploit i

This is great! I'd like to see a formal container security grade that works like: 1) Curate a list of all known (container) exploits 2) Run each exploit in environments of increasing security like permissions-based, jail, Docker and emulator 3) The percentage of prevented exploits would be the score from 0-100% Under this scheme, I'd expect naive attempts at containerization with permissions and jails to score around 0%, while Docker might be above 50% and Microsandbox could potentially reach 100%. This might satisfy some of our intuition around questions like "why not just use a jail?". Also the containers could run on a site on the open web as honeypots with cash or crypto prizes for pwning them to "prove" which containers achieve 100%. We might also need to redefine what "secure" means, since exploits like Rowhammer and Spectre may make nearly all conventional and cloud computing insecure. Or maybe it's a moving target, like how 64 bit encryption might have once been considered secure but now we need 128 bit or higher. Edit: the motivation behind this would be to find a container that's 100% secure without emulation, for performance and cost-savings benefits, as well as gaining insights into how to secure operating systems by containerizing their various services.

bjackman 1y ago
You cannot build a secure container runtime (against malicious containers) because underlying it is the Linux kernel. The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase. This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.
- Veserv 1y ago
  You cannot build a secure virtualization runtime because underlying it is the VMM. Until you have a secure VMM you are subject to precisely the same class of problems plaguing container runtimes. The only meaningful difference is that Linux containers target partitioning Linux kernel services which is a shared-by-default/default-allow environment that was never designed for and has never achieved meaningful security. The number of vulnerabilities resulting from, "whoopsie, we forgot to partition shared service 123" would be hilarious if it were not a complete lapse of security engineering in a product people are convinced is adequate for security-critical applications. Present a vulnerability assessment demonstrating a team of 10 with 3 years time (~10-30 M$, comparable to many commercially-motivated single-victim attacks these days) can find no vulnerabilities in your deployment or a formal proof of security and correctness otherwise we should stick with the default assumption that software if easily hacked instead of the extraordinary claim that demands extraordinary evidence.
  - nyrikki 1y ago
    While VMs do have an attack surface, it is vastly different than containers, which as you pointed out are not really a security system, but simply namespaces. Seacomp, capabilities, selinux, apparmor, etc.. can help harden containers, but most of the popular containers don't even drop root for services, and I was one of the people who tried to even get Docker/Moby etc.. to let you disable the privileged flag...which they refused to do. While some CRIs make this easier, any agent that can spin up a container should be considered a super user. With the docker --privlaged flag I could read the hosts root volume or even install efi bios files just using mknod etc, walking /sys to find the major/minor numbers. Namespaces are useful in a comprehensive security plan, but as you mentioned, they are not jails. It is true that both VMs and containers have attack surfaces, but the size of the attack surface on containers is much larger.
  - transpute 1y ago
    > You cannot build a secure virtualization runtime because underlying it is the VMM There are VMMs (e.g. pKVM in upstream Linux) with small SLoC that are isolated by silicon support for nested virtualization. This can be found on recent Google Pixel phones/tablets with strong isolation of untrusted Debian Arm Linux "Terminal" VM. A similar architecture was shipped a decade ago by Bromium and now on millions of HP business laptops, including hypervisor isolation of firmware, "Hypervisor Security : Lessons Learned — Ian Pratt, Bromium — Platform Security Summit 2018", https://www.youtube.com/watch?v=bNVe2y34dnM https://www.youtube.com/watch?v=bNVe2y34dnM Christian Slater, HP cybersecurity ("Wolf") edutainment on nested virt hypervisor in printers, https://www.youtube.com/watch?v=DjMSq3n3Gqs https://www.youtube.com/watch?v=DjMSq3n3Gqs
    - delusional 1y ago
      > silicon support for nested virtualization Is there any guarantee that this "silicon support" is any safer than the software? Once we break the software abstraction down far enough it's all just configuring hardware. Conversely, once you start baking significant complexity into hardware (such as strong security boundaries) it would seem like hardware would be subject to exactly the same bugs as software would, except it will be hard to update of course.
- akdev1l 1y ago
  One can definitely build a container runtime that uses virtualization to protect the host For example there is Kata containers https://katacontainers.io/ https://katacontainers.io/ This can be used with regular `podman` by just changing the container runtime so there’s no even need for any extra tooling In theory you could shove the container runtime into something like k8s
  - bjackman 1y ago
    > container runtime that uses virtualization to protect the host True, by "container" I really meant "shared-kernel container". > In theory you could shove the container runtime into something like k8s Yeah this is actually supported by k8s. Whether that means it's actually reasonable to run completely untrusted workloads on your own cluster is another question. But it definitely seems like a really good defense-in-depth feature.
- ignoramous 1y ago
  > ... drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value ... Depends I guess as Android has had quite a bit of success with seccomp-bpf & Android-specific flavour of SELinux [0] > Until we have a properly hardened and memory safe OS ... faster than running MicroVMs on a Linux host. Andy Tanenbaum might say, Micro Kernels would do just as well. [0] https://youtu.be/WxbOq8IGEiE https://youtu.be/WxbOq8IGEiE
  - lillecarl 1y ago
    You also have gVisor, which runs all syscall through some Go history that's supposedly safe enough for Google.
    - bjackman 1y ago
      gVisor uses virtualization
  - bjackman 1y ago
    > Android Exactly. Android pulls this off by being extremely constrained. It's dramatically less flexible than an OCI runtime. If you wanna run a random unenlightened workload on it you're probably gonna have a hard time. > Micro Kernels would do just as well. Yea this goes in the right direction. In the end a lot of kernel work I look at is basically about trying to retrofit benefits of microkernels onto Linux. Saying "we should just use an actual microkernel" is a bit like "Russia and Ukraine should just make peace" IMO though.
tptacek 1y ago
The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.
- delusional 1y ago
  > it's just sort of understood that a kernel LPE is going to break containers. I think it's generally understood that any sort of kernel LPE can potentially (and therefore is generally considered to) lead to breaking all security boundaries on the local machine, since the kernel contains no internal security boundaries. That includes both containers, but also everything else such a user separation, hardware virtualization controlled by the local kernel, and kernel private secrets.
  - transpute 1y ago
    > hardware virtualization controlled by the local kernel In some architectures, kernel LPE does not break platform (L0/EL2) virtualization, https://news.ycombinator.com/item?id=44141164 https://news.ycombinator.com/item?id=44141164 L0/EL2 L1/EL1 pKVM KVM AX Hyper-V / Xen / ESX
    - tptacek 1y ago
      Most Linux kernel LPEs --- in fact, the overwhelming majority of them --- don't threaten KVM hosts when exploited in KVM guests.
      - billywhizz 1y ago
        is there anything good written up on this?
        1y ago
Etheryte 1y ago
In a way, containers already run as honeypots with cash or crypto prizes, it's called production code and plenty of people are looking for holes day and night. While this setup sounds like a nice idea conceptually, the monetary incentives it could offer would surely be miniscule compared to real targets.
godelski 1y ago
Importantly I'd like to see the configurations of the machines. There's a lot you can do to docker or systemd spawns that greatly vary the security levels. This would really help show what needs to be done and what configurations lead to what risks. Basically I'd love to see a giant ablation

bjackman 1y ago

You cannot build a secure container runtime (against malicious containers) because underlying it is the Linux kernel. The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase. This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.

Veserv 1y ago
You cannot build a secure virtualization runtime because underlying it is the VMM. Until you have a secure VMM you are subject to precisely the same class of problems plaguing container runtimes. The only meaningful difference is that Linux containers target partitioning Linux kernel services which is a shared-by-default/default-allow environment that was never designed for and has never achieved meaningful security. The number of vulnerabilities resulting from, "whoopsie, we forgot to partition shared service 123" would be hilarious if it were not a complete lapse of security engineering in a product people are convinced is adequate for security-critical applications. Present a vulnerability assessment demonstrating a team of 10 with 3 years time (~10-30 M$, comparable to many commercially-motivated single-victim attacks these days) can find no vulnerabilities in your deployment or a formal proof of security and correctness otherwise we should stick with the default assumption that software if easily hacked instead of the extraordinary claim that demands extraordinary evidence.
- nyrikki 1y ago
  While VMs do have an attack surface, it is vastly different than containers, which as you pointed out are not really a security system, but simply namespaces. Seacomp, capabilities, selinux, apparmor, etc.. can help harden containers, but most of the popular containers don't even drop root for services, and I was one of the people who tried to even get Docker/Moby etc.. to let you disable the privileged flag...which they refused to do. While some CRIs make this easier, any agent that can spin up a container should be considered a super user. With the docker --privlaged flag I could read the hosts root volume or even install efi bios files just using mknod etc, walking /sys to find the major/minor numbers. Namespaces are useful in a comprehensive security plan, but as you mentioned, they are not jails. It is true that both VMs and containers have attack surfaces, but the size of the attack surface on containers is much larger.
- transpute 1y ago
  > You cannot build a secure virtualization runtime because underlying it is the VMM There are VMMs (e.g. pKVM in upstream Linux) with small SLoC that are isolated by silicon support for nested virtualization. This can be found on recent Google Pixel phones/tablets with strong isolation of untrusted Debian Arm Linux "Terminal" VM. A similar architecture was shipped a decade ago by Bromium and now on millions of HP business laptops, including hypervisor isolation of firmware, "Hypervisor Security : Lessons Learned — Ian Pratt, Bromium — Platform Security Summit 2018", https://www.youtube.com/watch?v=bNVe2y34dnM https://www.youtube.com/watch?v=bNVe2y34dnM Christian Slater, HP cybersecurity ("Wolf") edutainment on nested virt hypervisor in printers, https://www.youtube.com/watch?v=DjMSq3n3Gqs https://www.youtube.com/watch?v=DjMSq3n3Gqs
  - delusional 1y ago
    > silicon support for nested virtualization Is there any guarantee that this "silicon support" is any safer than the software? Once we break the software abstraction down far enough it's all just configuring hardware. Conversely, once you start baking significant complexity into hardware (such as strong security boundaries) it would seem like hardware would be subject to exactly the same bugs as software would, except it will be hard to update of course.
akdev1l 1y ago
One can definitely build a container runtime that uses virtualization to protect the host For example there is Kata containers https://katacontainers.io/ https://katacontainers.io/ This can be used with regular `podman` by just changing the container runtime so there’s no even need for any extra tooling In theory you could shove the container runtime into something like k8s
- bjackman 1y ago
  > container runtime that uses virtualization to protect the host True, by "container" I really meant "shared-kernel container". > In theory you could shove the container runtime into something like k8s Yeah this is actually supported by k8s. Whether that means it's actually reasonable to run completely untrusted workloads on your own cluster is another question. But it definitely seems like a really good defense-in-depth feature.
ignoramous 1y ago
> ... drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value ... Depends I guess as Android has had quite a bit of success with seccomp-bpf & Android-specific flavour of SELinux [0] > Until we have a properly hardened and memory safe OS ... faster than running MicroVMs on a Linux host. Andy Tanenbaum might say, Micro Kernels would do just as well. [0] https://youtu.be/WxbOq8IGEiE https://youtu.be/WxbOq8IGEiE
- lillecarl 1y ago
  You also have gVisor, which runs all syscall through some Go history that's supposedly safe enough for Google.
  - bjackman 1y ago
    gVisor uses virtualization
- bjackman 1y ago
  > Android Exactly. Android pulls this off by being extremely constrained. It's dramatically less flexible than an OCI runtime. If you wanna run a random unenlightened workload on it you're probably gonna have a hard time. > Micro Kernels would do just as well. Yea this goes in the right direction. In the end a lot of kernel work I look at is basically about trying to retrofit benefits of microkernels onto Linux. Saying "we should just use an actual microkernel" is a bit like "Russia and Ukraine should just make peace" IMO though.

tptacek 1y ago

The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.

delusional 1y ago
> it's just sort of understood that a kernel LPE is going to break containers. I think it's generally understood that any sort of kernel LPE can potentially (and therefore is generally considered to) lead to breaking all security boundaries on the local machine, since the kernel contains no internal security boundaries. That includes both containers, but also everything else such a user separation, hardware virtualization controlled by the local kernel, and kernel private secrets.
- transpute 1y ago
  > hardware virtualization controlled by the local kernel In some architectures, kernel LPE does not break platform (L0/EL2) virtualization, https://news.ycombinator.com/item?id=44141164 https://news.ycombinator.com/item?id=44141164 L0/EL2 L1/EL1 pKVM KVM AX Hyper-V / Xen / ESX
  - tptacek 1y ago
    Most Linux kernel LPEs --- in fact, the overwhelming majority of them --- don't threaten KVM hosts when exploited in KVM guests.
    - billywhizz 1y ago
      is there anything good written up on this?
      - 1y ago

Etheryte 1y ago

In a way, containers already run as honeypots with cash or crypto prizes, it's called production code and plenty of people are looking for holes day and night. While this setup sounds like a nice idea conceptually, the monetary incentives it could offer would surely be miniscule compared to real targets.

godelski 1y ago

Importantly I'd like to see the configurations of the machines. There's a lot you can do to docker or systemd spawns that greatly vary the security levels. This would really help show what needs to be done and what configurations lead to what risks. Basically I'd love to see a giant ablation