5 ms·

Regarding your first link I had questions about the following passages from it: >"People like to say “Docker isn’t a security boundary”, but that’s not so true

Regarding your first link I had questions about the following passages from it: >"People like to say “Docker isn’t a security boundary”, but that’s not so true anymore, though it once was." Could you explain why that wasn't true and why it is now? >"Systems security people spent almost a decade dunking on Docker because of all the gaps in this simplified container model. But nobody really runs containers like this anymore." Could you elaborate? People don't run container like what? It isn't clear from the preceding paragraph.

tptacek 6y ago
Modern Docker running in default configurations is reasonably well locked down. I wouldn't use it for multitenant scenarios, though.
- varun_chopra 6y ago
  Can you elaborate why? Would love to know where it falls behind and how to fix these without going the Firecracker route.
  - steventhedev 6y ago
    With containers, both the kernel and the hypervisor are shared. With vms, only the hypervisor is shared. It's a matter of having a smaller attack surface. There are plenty of container images that run with root access by default, which is almost full access to the kernel. This means that if the application running in the container is compromised, you need to rely on the kernel enforcing the sandbox between containers. This is a relatively new threat (root not being fully trusted), so beyond there simply being more attack surface, there's likely to be more bugs/vulns out there to be discovered. With effort and care you can safely run this but reducing attack surface is a good idea for defense in depth.
    - fpoling 6y ago
      If one only allows container to run as a non-root user (no user namespace either) with all privileges dropped with strong mount isolation and some form of syscall filtering, then the attack surface is similar to that of hypervisors if not smaller while the performance is significantly better. But yes, quite a few services assumes they have root privileges and do not work as is in such containers, like recent OpenSSH. For those cases VM isolation makes for much smaller attack surface.
      - tptacek 6y ago
        The security models really aren't comparable. Again, see the blog post, which offers two examples of attacks that break the model you're proposing. I don't want to get into too much detail in this thread because I really just wanted to add some data to questions Julia Evans specifically asked in her post.
        fpoling 6y ago
        The blog post incorrectly states that Go is a memory-safe language. It is not in its standard configuration. And it does not mention that hypervisor bugs allowing to escape from a VM are typically due to wrong logic, not memory safety. Using memory-safe language does not protect from those. And that in turn is the reflection of complexity of hypervisor interfaces in modern CPU. And with VM one gets much greater exposure to hardware bugs as VM has access to more instructions.
        tptacek 6y ago
        Homer-into-the-hedges.gif.
  - tptacek 6y ago
    The blog post linked upthread goes into some detail about why; answering that question is kind of the point of the post.
iampims 6y ago
Not OP but most public cloud need stronger isolation than what Docker provides. GKE uses gVisor, AWS uses Firecracker now. When they started, lambda code was scheduled on a different EC2 instance per customer. VM isolation is just so much stronger. Firecracker hopes to make launching VMs as quick as launching containers while retaining the great isolation benefits of a VM.
- ignoramous 6y ago
  > ...GKE uses gVisor Another commentator on this thread points out that GKE uses trimmed-down Chromium OS? https://news.ycombinator.com/item?id=25884851 https://news.ycombinator.com/item?id=25884851 Edit: I guess container-optimized Chromium OS and gVisor are complementary (that is, you're welcome to run your app in gVisor sandboxed containers on a guest VM running Google's container-optimized OS). Ref: https://go-review.googlesource.com/c/playground/+/195983 https://go-review.googlesource.com/c/playground/+/195983
  - tptacek 6y ago
    See, for instance, https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes...
    - ignoramous 6y ago
      Thanks. I remember mrkurt say fly.io evaluated gVisor but settled on Firecracker eventually, though, I must ask: can gVisor on its own achieve same level of protection as micro-VMs? Even if not, can the protection offered by application kernels like gVisor and Kata be considered enough for multi-tenant workloads (like the one fly.io runs) inside the same guest (VM)? Also, if I may, in what cases would one prefer to use Firecracker managed micro-VM running gVisor sandboxed containers? I'd imagine gVisor slows things down, so it might not be for everyone, but I'm curious as to what value-add might make gVisor worth that.
      - 6y ago

tptacek 6y ago

Modern Docker running in default configurations is reasonably well locked down. I wouldn't use it for multitenant scenarios, though.

varun_chopra 6y ago
Can you elaborate why? Would love to know where it falls behind and how to fix these without going the Firecracker route.
- steventhedev 6y ago
  With containers, both the kernel and the hypervisor are shared. With vms, only the hypervisor is shared. It's a matter of having a smaller attack surface. There are plenty of container images that run with root access by default, which is almost full access to the kernel. This means that if the application running in the container is compromised, you need to rely on the kernel enforcing the sandbox between containers. This is a relatively new threat (root not being fully trusted), so beyond there simply being more attack surface, there's likely to be more bugs/vulns out there to be discovered. With effort and care you can safely run this but reducing attack surface is a good idea for defense in depth.
  - fpoling 6y ago
    If one only allows container to run as a non-root user (no user namespace either) with all privileges dropped with strong mount isolation and some form of syscall filtering, then the attack surface is similar to that of hypervisors if not smaller while the performance is significantly better. But yes, quite a few services assumes they have root privileges and do not work as is in such containers, like recent OpenSSH. For those cases VM isolation makes for much smaller attack surface.
    - tptacek 6y ago
      The security models really aren't comparable. Again, see the blog post, which offers two examples of attacks that break the model you're proposing. I don't want to get into too much detail in this thread because I really just wanted to add some data to questions Julia Evans specifically asked in her post.
      - fpoling 6y ago
        The blog post incorrectly states that Go is a memory-safe language. It is not in its standard configuration. And it does not mention that hypervisor bugs allowing to escape from a VM are typically due to wrong logic, not memory safety. Using memory-safe language does not protect from those. And that in turn is the reflection of complexity of hypervisor interfaces in modern CPU. And with VM one gets much greater exposure to hardware bugs as VM has access to more instructions.
        tptacek 6y ago
        Homer-into-the-hedges.gif.
- tptacek 6y ago
  The blog post linked upthread goes into some detail about why; answering that question is kind of the point of the post.

iampims 6y ago

Not OP but most public cloud need stronger isolation than what Docker provides. GKE uses gVisor, AWS uses Firecracker now. When they started, lambda code was scheduled on a different EC2 instance per customer. VM isolation is just so much stronger. Firecracker hopes to make launching VMs as quick as launching containers while retaining the great isolation benefits of a VM.

ignoramous 6y ago
> ...GKE uses gVisor Another commentator on this thread points out that GKE uses trimmed-down Chromium OS? https://news.ycombinator.com/item?id=25884851 https://news.ycombinator.com/item?id=25884851 Edit: I guess container-optimized Chromium OS and gVisor are complementary (that is, you're welcome to run your app in gVisor sandboxed containers on a guest VM running Google's container-optimized OS). Ref: https://go-review.googlesource.com/c/playground/+/195983 https://go-review.googlesource.com/c/playground/+/195983
- tptacek 6y ago
  See, for instance, https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes...
  - ignoramous 6y ago
    Thanks. I remember mrkurt say fly.io evaluated gVisor but settled on Firecracker eventually, though, I must ask: can gVisor on its own achieve same level of protection as micro-VMs? Even if not, can the protection offered by application kernels like gVisor and Kata be considered enough for multi-tenant workloads (like the one fly.io runs) inside the same guest (VM)? Also, if I may, in what cases would one prefer to use Firecracker managed micro-VM running gVisor sandboxed containers? I'd imagine gVisor slows things down, so it might not be for everyone, but I'm curious as to what value-add might make gVisor worth that.
    - 6y ago