4 ms·

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots m

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on steroids. This architecture allows it to provide a flexible resource footprint (i.e. one based on threads and memory mappings, not fixed guest physical resources) while also lowering the fixed costs of virtualization. However, this comes at the price of reduced application compatibility and higher per-system call overhead. From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). So, you have pretty much complete control over what your apps can do. This for me, is sort of the key takeway from the blog post too: "because we use gVisor to increase the security of Google's own internal workloads, it continuously benefits from our expertise and experience running containers at scale in a security-first environment". So, Google's using something like this internally too for their own workloads, which should be a pretty good sign this works in real life. [1] https://gvisor.dev/docs/architecture_guide/ https://gvisor.dev/docs/architecture_guide/ [2] https://github.com/google/gvisor https://github.com/google/gvisor [3] https://gvisor.dev/ https://gvisor.dev/

roryrjb 7y ago
Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.
- WestCoastJustin 7y ago
  > re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...
  - lima 7y ago
    > Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
    - WestCoastJustin 7y ago
      Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
      - lima 7y ago
        gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
      - raesene9 7y ago
        So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do. Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...
thesandlord 7y ago
> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contract#sandbox https://cloud.google.com/run/docs/reference/container-contra...
- WestCoastJustin 7y ago
  Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative
  - spyspy 7y ago
    The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
    - cameronbrown 7y ago
      > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
      - deleted 7y ago
        [deleted]
      - asciimike 7y ago
        Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
prattmic 7y ago
> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy configuration for your application; just put your application inside gVisor and go. These are significant differences over simply using something like seccomp on its own (what the architecture guide calls "Rule-based execution"). Some of this is covered in our security model: https://gvisor.dev/docs/architecture_guide/security/#principles-defense-in-depth https://gvisor.dev/docs/architecture_guide/security/#princip...
- WestCoastJustin 7y ago
  Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too (https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392).
  - bradfitz 7y ago
    App Engine now uses gVisor too: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
- saagarjha 7y ago
  Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?
  - prattmic 7y ago
    For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2]. > how do you implement features that ostensibly require calls to the OS anyways? gVisor's kernel is a user-space program, so it can and does make system calls to the host OS. Some examples: * An application blocks trying to read(2) from a pipe. gVisor ultimately implements blocking by waiting on a Go channel. The Go runtime will ultimately implement this with a futex(2) call to the host OS. * An application reads from a file that is ultimately backed by a file on the host (provided by the Gofer [3]). This will result in a pread(2) system call to the host. The purpose here isn't to avoid the host completely (that's not possible), but to limit exposure to the host. gVisor can implement all the parts of Linux it does on a much smaller subset of host system calls. Anything we don't use is blocked by a second-level seccomp sandbox around the kernel. e.g., the kernel cannot make obscure system calls, or even open files or create sockets on the host (those operations are controlled by an external agent). [1] https://github.com/google/gvisor/tree/master/test/syscalls/linux https://github.com/google/gvisor/tree/master/test/syscalls/l... [2] https://github.com/google/syzkaller https://github.com/google/syzkaller [3] https://gvisor.dev/docs/architecture_guide/overview/ https://gvisor.dev/docs/architecture_guide/overview/
equalunique 7y ago
My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s https://www.youtube.com/watch?v=Ur0hbW_K66s I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM https://www.youtube.com/watch?v=HgtRAbE1nBM
blaisio 7y ago
It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did). This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really just in a big technology cycle. Dynamically typed -> statically typed -> dynamically typed; bare metal -> API wrappers -> VMs -> containers -> API wrappers. Soon we'll probably be back to bare metal.
- ryacko 7y ago
  There are web hosts offer Raspberry PIs, but they tend to be more expensive than VMs. I'm guessing colocation costs are dominant.

roryrjb 7y ago

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.

WestCoastJustin 7y ago
> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...
- lima 7y ago
  > Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
  - WestCoastJustin 7y ago
    Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
    - lima 7y ago
      gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
    - raesene9 7y ago
      So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do. Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...

thesandlord 7y ago

> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contract#sandbox https://cloud.google.com/run/docs/reference/container-contra...

WestCoastJustin 7y ago
Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative
- spyspy 7y ago
  The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
  - cameronbrown 7y ago
    > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
    - deleted 7y ago
      [deleted]
    - asciimike 7y ago
      Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

prattmic 7y ago

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy configuration for your application; just put your application inside gVisor and go. These are significant differences over simply using something like seccomp on its own (what the architecture guide calls "Rule-based execution"). Some of this is covered in our security model: https://gvisor.dev/docs/architecture_guide/security/#principles-defense-in-depth https://gvisor.dev/docs/architecture_guide/security/#princip...

WestCoastJustin 7y ago
Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too (https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392).
- bradfitz 7y ago
  App Engine now uses gVisor too: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
saagarjha 7y ago
Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?
- prattmic 7y ago
  For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2]. > how do you implement features that ostensibly require calls to the OS anyways? gVisor's kernel is a user-space program, so it can and does make system calls to the host OS. Some examples: * An application blocks trying to read(2) from a pipe. gVisor ultimately implements blocking by waiting on a Go channel. The Go runtime will ultimately implement this with a futex(2) call to the host OS. * An application reads from a file that is ultimately backed by a file on the host (provided by the Gofer [3]). This will result in a pread(2) system call to the host. The purpose here isn't to avoid the host completely (that's not possible), but to limit exposure to the host. gVisor can implement all the parts of Linux it does on a much smaller subset of host system calls. Anything we don't use is blocked by a second-level seccomp sandbox around the kernel. e.g., the kernel cannot make obscure system calls, or even open files or create sockets on the host (those operations are controlled by an external agent). [1] https://github.com/google/gvisor/tree/master/test/syscalls/linux https://github.com/google/gvisor/tree/master/test/syscalls/l... [2] https://github.com/google/syzkaller https://github.com/google/syzkaller [3] https://gvisor.dev/docs/architecture_guide/overview/ https://gvisor.dev/docs/architecture_guide/overview/

equalunique 7y ago

My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s https://www.youtube.com/watch?v=Ur0hbW_K66s I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM https://www.youtube.com/watch?v=HgtRAbE1nBM

blaisio 7y ago

It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did). This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really just in a big technology cycle. Dynamically typed -> statically typed -> dynamically typed; bare metal -> API wrappers -> VMs -> containers -> API wrappers. Soon we'll probably be back to bare metal.

ryacko 7y ago
There are web hosts offer Raspberry PIs, but they tend to be more expensive than VMs. I'm guessing colocation costs are dominant.