6 ms·

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.

WestCoastJustin 7y ago
> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...
- lima 7y ago
  > Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
  - WestCoastJustin 7y ago
    Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
    - lima 7y ago
      gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
    - raesene9 7y ago
      So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do. Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...
      - windexh8er 7y ago
        > Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that... There are a bunch of legitimate reasons to run Docker in Docker. The most obvious is in a build pipeline. For example Jenkins does Docker builds in containers all the time.
  - ian-lewis 7y ago
    Rather that the contain/don't-contain dichotomy, what's more important is gVisor's design principle that it always has 2 layers of isolation from the host and doesn't rely on any one bug in the Linux kernel, sentry, or elsewhere in order to break out of the sandbox. This leaves you less exposed to 0-day attacks and lags in patching kernels. You can't get that from normal Linux containers due to their fundamental design.
- 013a 7y ago
  That's fair, but at the same time: If the end-state is "containers should contain, they're secure, any insecurities are bugs" then why do we see so many defense-in-depth strategies like gVisor pop up which provide legitimate value to consumers? At what point are we just reinventing the VM hypervisor, but worse because every single one of these systems already has a VM hypervisor running somewhere? It seems likely to me that in the not-so-distant future the "Container" terminology won't actually mean anything because we'll figure out the engineering difficulty behind merging the best parts of VMs with the best parts of Containers, and managed systems like Fargate or even GKE don't really need both a VM hypervisor and a Container hypervisor when they're so similar.
  - lima 7y ago
    gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.
    - wahern 7y ago
      Memory overcommit is addressed by virtio memory ballooning (https://www.linux-kvm.org/page/Projects/auto-ballooning https://www.linux-kvm.org/page/Projects/auto-ballooning). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs (https://virtio-fs.gitlab.io/ https://virtio-fs.gitlab.io/) is just about ready to roll. There's still the issue that you're running an entire extra kernel. Not sure that's much slower than using Go; it's probably faster if what was described about bouncing on futexes elsethread is true. gVisor sounds like the kind of solution that makes sense for Google but not something that would survive in the wider community. The concept sounds great, but using Go sounds horrible, though I'm sure Go made prototyping the concept super simple--specifically goroutines reify execution flow in a nice way, but so would stackful coroutines in C or even Rust, which is easy to implement if you don't need to worry about deep recursion.
      - pjmlp 7y ago

WestCoastJustin 7y ago

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...

lima 7y ago
> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
- WestCoastJustin 7y ago
  Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
  - lima 7y ago
    gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
  - raesene9 7y ago
    So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do. Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...
    - windexh8er 7y ago
      > Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that... There are a bunch of legitimate reasons to run Docker in Docker. The most obvious is in a build pipeline. For example Jenkins does Docker builds in containers all the time.
- ian-lewis 7y ago
  Rather that the contain/don't-contain dichotomy, what's more important is gVisor's design principle that it always has 2 layers of isolation from the host and doesn't rely on any one bug in the Linux kernel, sentry, or elsewhere in order to break out of the sandbox. This leaves you less exposed to 0-day attacks and lags in patching kernels. You can't get that from normal Linux containers due to their fundamental design.
013a 7y ago
That's fair, but at the same time: If the end-state is "containers should contain, they're secure, any insecurities are bugs" then why do we see so many defense-in-depth strategies like gVisor pop up which provide legitimate value to consumers? At what point are we just reinventing the VM hypervisor, but worse because every single one of these systems already has a VM hypervisor running somewhere? It seems likely to me that in the not-so-distant future the "Container" terminology won't actually mean anything because we'll figure out the engineering difficulty behind merging the best parts of VMs with the best parts of Containers, and managed systems like Fargate or even GKE don't really need both a VM hypervisor and a Container hypervisor when they're so similar.
- lima 7y ago
  gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.
  - wahern 7y ago
    Memory overcommit is addressed by virtio memory ballooning (https://www.linux-kvm.org/page/Projects/auto-ballooning https://www.linux-kvm.org/page/Projects/auto-ballooning). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs (https://virtio-fs.gitlab.io/ https://virtio-fs.gitlab.io/) is just about ready to roll. There's still the issue that you're running an entire extra kernel. Not sure that's much slower than using Go; it's probably faster if what was described about bouncing on futexes elsethread is true. gVisor sounds like the kind of solution that makes sense for Google but not something that would survive in the wider community. The concept sounds great, but using Go sounds horrible, though I'm sure Go made prototyping the concept super simple--specifically goroutines reify execution flow in a nice way, but so would stackful coroutines in C or even Rust, which is easy to implement if you don't need to worry about deep recursion.
    - pjmlp 7y ago