11 ms·

GKE Sandbox: Independent operating system kernel to each container

158 points by alpb 7y ago

WestCoastJustin 7y ago
For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on steroids. This architecture allows it to provide a flexible resource footprint (i.e. one based on threads and memory mappings, not fixed guest physical resources) while also lowering the fixed costs of virtualization. However, this comes at the price of reduced application compatibility and higher per-system call overhead. From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). So, you have pretty much complete control over what your apps can do. This for me, is sort of the key takeway from the blog post too: "because we use gVisor to increase the security of Google's own internal workloads, it continuously benefits from our expertise and experience running containers at scale in a security-first environment". So, Google's using something like this internally too for their own workloads, which should be a pretty good sign this works in real life. [1] https://gvisor.dev/docs/architecture_guide/ https://gvisor.dev/docs/architecture_guide/ [2] https://github.com/google/gvisor https://github.com/google/gvisor [3] https://gvisor.dev/ https://gvisor.dev/
- roryrjb 7y ago
  Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.
  - WestCoastJustin 7y ago
    > re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...
    - lima 7y ago
      > Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
      - WestCoastJustin 7y ago
        Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
        lima 7y ago
        gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
- thesandlord 7y ago
  > So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contract#sandbox https://cloud.google.com/run/docs/reference/container-contra...
  - WestCoastJustin 7y ago
    Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative
    - spyspy 7y ago
      The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
      - cameronbrown 7y ago
        > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
        deleted 7y ago
        [deleted]
        asciimike 7y ago
        Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
- prattmic 7y ago
  > From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy configuration for your application; just put your application inside gVisor and go. These are significant differences over simply using something like seccomp on its own (what the architecture guide calls "Rule-based execution"). Some of this is covered in our security model: https://gvisor.dev/docs/architecture_guide/security/#principles-defense-in-depth https://gvisor.dev/docs/architecture_guide/security/#princip...
  - WestCoastJustin 7y ago
    Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too (https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392).
    - bradfitz 7y ago
      App Engine now uses gVisor too: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
  - saagarjha 7y ago
    Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?
    - prattmic 7y ago
      For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2]. > how do you implement features that ostensibly require calls to the OS anyways? gVisor's kernel is a user-space program, so it can and does make system calls to the host OS. Some examples: * An application blocks trying to read(2) from a pipe. gVisor ultimately implements blocking by waiting on a Go channel. The Go runtime will ultimately implement this with a futex(2) call to the host OS. * An application reads from a file that is ultimately backed by a file on the host (provided by the Gofer [3]). This will result in a pread(2) system call to the host. The purpose here isn't to avoid the host completely (that's not possible), but to limit exposure to the host. gVisor can implement all the parts of Linux it does on a much smaller subset of host system calls. Anything we don't use is blocked by a second-level seccomp sandbox around the kernel. e.g., the kernel cannot make obscure system calls, or even open files or create sockets on the host (those operations are controlled by an external agent). [1] https://github.com/google/gvisor/tree/master/test/syscalls/linux https://github.com/google/gvisor/tree/master/test/syscalls/l... [2] https://github.com/google/syzkaller https://github.com/google/syzkaller [3] https://gvisor.dev/docs/architecture_guide/overview/ https://gvisor.dev/docs/architecture_guide/overview/
- equalunique 7y ago
  My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s https://www.youtube.com/watch?v=Ur0hbW_K66s I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM https://www.youtube.com/watch?v=HgtRAbE1nBM
- blaisio 7y ago
  It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did). This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really just in a big technology cycle. Dynamically typed -> statically typed -> dynamically typed; bare metal -> API wrappers -> VMs -> containers -> API wrappers. Soon we'll probably be back to bare metal.
  - ryacko 7y ago
    There are web hosts offer Raspberry PIs, but they tend to be more expensive than VMs. I'm guessing colocation costs are dominant.
conroy 7y ago
For those more familiar with Kubernetes and gVisor, would this allow me to build a CI/CD service that runs untrusted user code?
- snug 7y ago
  > gVisor provides a virtualized environment in order to sandbox untrusted containers. So yes
- raesene9 7y ago
  Well like all things in security, that kind of depends :) What gVisor does is provide a smaller attack surface to a containerized process, when compared with a "traditional" Docker container using standard Docker setup (you can, of course harden Docker containers considerably from base, if you are so inclined). However it doesn't affect anything outside of that interface so, for example, if your CI/CD process is running on a network that has other insecure services on them, then gVisor alone won't really help you if malicious code is executed inside a container allowing an attacker to start probing the environment from the perspective of that container.
raesene9 7y ago
This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm really not a fan of the opening line where it references the old trope of "containers don't contain" The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. Have there been vulns that allow for container breakout, sure there have, but every piece of software (including gVisor) has had vulnerabilities in it. What you can say about gVisor is that it likely presents a smaller attack surface in its default configuration than a runc style Docker container. However, of course, there's nothing to stop people tightening up on the defaults and still using runc. As an aside for anyone who thinks container breakouts are trivially easy, you can go to https://contained.af https://contained.af and win yourself some money :)
- longtermsec 7y ago
  The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. -- not just being contrarian here, actually, the reality is that it might be trivial. and it was demonstratively trivial for a long time (see CVE-2019-5736) As for contained.af -- its not a good indicator, it mostly indicates that the reward doesnt meet the market price for demonstrating an escape from a set of hardened namespaces (which is going to cost more than an escape from "any docker container").
  - raesene9 7y ago
    So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I was talking about, all software has vulns, including runc, including gvisor. Stating that "containers don't contain" implies that it's not just a specific bug, but that architecturally the process is flawed (at least IMHO), which I would suggest is at the least an over-simplification. as to contained.af, well if it was indeed "trivial" then surely not a large reward would be required :)
bogomipz 7y ago
I have read that all containers at Google run inside of a VM and indeed that article mentions that gVisor is in use in things like App Engine and their internal workloads. So if containers on GKE were already being spun up inside lightweights VMs what does allowing customer's to select the gVisor runtime offer beyond whatever Google's existing lightweight VM already provides?
- thesandlord 7y ago
  Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.
  - bogomipz 7y ago
    What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1] Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around: https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13 [1] https://queue.acm.org/detail.cfm?id=2898444 https://queue.acm.org/detail.cfm?id=2898444
    - yoshiat 7y ago
      (Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases. [1] https://www.youtube.com/watch?v=TQfc8OlB2sg https://www.youtube.com/watch?v=TQfc8OlB2sg
muricula 7y ago
GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/ https://gvisor.dev/docs/architecture_guide/performance/
- lifty 7y ago
  where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.
  - muricula 7y ago
    Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.
    - lifty 7y ago
      Missed that one. The gVisor calls are closer to 40k but the difference is indeed big.
    - amscanne 7y ago
      It's just log, not log log. The chart generates from .csv hosted on the site [1] and the benchmark tools are all open. The ptrace numbers are 20x and the KVM platform is actually lower than the Docker default case (though that doesn't mean everything is faster, as system call time is only one factor). As I note above, I think you're confused about what the KVM platform is -- it's not a VM. [1] https://gvisor.dev/performance/syscall.csv https://gvisor.dev/performance/syscall.csv
- danbeaulieu 7y ago
whalesalad 7y ago
Isn't this server-side react rendering? What are we doing? We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something? Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technology with the ergonomics of containers and then we'll be WAY closer to true portability and better security.
- lemoncucumber 7y ago
  One reason I can see is the same reason that linuxkit [1] is a really interesting way of putting together Linux OS images: the container ecosystem has produced some tools that are really useful for building and packaging up Linux userspaces, and being able to reuse those tools in other ways is valuable. With linuxkit you give up some of the niceties of image layer caching, but with an approach like this you get the best of both worlds -- the isolation of VMs but the tooling and usability of containers. [1] https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit
- amscanne 7y ago
  (I'm a co-author of the blog post) Many functions of the kernel are still effectively shared: memory management (e.g. reclaim, swap), thread scheduling, etc. The application is simply limited in its ability to interact with the shared kernel, and functionality related to system APIs is isolated. Arguably I think this is closer to the ergonomics of containers, but with compatibility and performance trade-offs.
- dastbe 7y ago
  (I work at AWS) Have you seen https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... ? The team here is working to make firecracker as seamless as possible for running containerized applications in a microvm.
ronsor 7y ago
At this point, why not just use a virtual machine? We've come full circle!
- jacques_chester 7y ago
  Mostly the performance characteristics. A virtual machine presenting as a machine needs an operating system to be useful. Most operating systems have long-engrained assumptions about the nature of the world, such as: "There is a time when I go from power-off to power-on, and it is rare, so I may perform expensive operations then to amortise their cost over running time". or "While running, time does not skip and hardware does not change". The practical upshot being that the OS needs to be booted from scratch in a number of scenarios. But it's not the OS that provides value. It's a means to an end, and that end is to run software. Most software written to run on OSes also have engrained assumptions, such as "I will come to be launched on a fully-booted system". Containers move the virtualisation up from hardware to the OS API surface. Because the cost of booting is now amortised over all containers running on the system, the original assumptions of both OS designers and software designers become, approximately, true again. So you're right, we came full circle, but not to a point that means "use fully-dressed VMs again".
andrewstuart 7y ago
Sandboxed containers with kernels - so what's the difference now between this and fully isolated virtual machine? Another approach might be to make virtual machine technology more like containers. Then the two shall meet.
- edoo 7y ago
  I didn't dig into the implementation details but the term para-hvm came to mind, not quite para virtual but not quite full hvm. Perhaps if security is a real issue then HVM is the only real choice.
metta2uall 7y ago
I quite like this defense-in-depth approach, but it's disappointing that it will only be available as part of the probably expensive GKE Advanced. I would have thought safety features should be standard..
- dilyevsky 7y ago
  I think either way control plane is free now?
  - metta2uall 7y ago
    Well gVisor doesn't use the control plane. It is free, but I wouldn't think it has a high cpu or memory load, and Google would make a lot of profit on the nodes.
    - dilyevsky 7y ago
      I know but they may conceivably just charge fixed fee for enabling that option on the nodepool. > it has a high cpu or memory load, and Google would make a lot of profit on the nodes. They currently solve that problem by having their node VMs melt down at like 50% utilization so you have to run everything with huge padding.
fulafel 7y ago
gVisor is easy to run in on your local dev laptop Docker too. It's a nice alternative to running Docker in a VM, if you prefer a security boundary between random Docker containers you get off Docker Hub and your host machine. After you built or downloaded the gVisor single binary to /usr/local/bin/ or wherever, just put a snippet provided in the gVisor README in the Docker settings file ("runtime": {...}), and bob's your uncle.
nullwasamistake 7y ago
The only advantage I see to containers over VM is RAM sharing. Beyond that, hardware VM's are better performing and much more secure. gVisor is just another flavor of containers that replaces kernel interfaces with a Go shim layer to reduce the attack surface in return for worse performance. If somebody could hack ram sharing/overcommit into traditional VM's all this container nonsense could be dispensed with. Containers are a virtualization layer just like the old days when we used the JVM to run "safe" applets on client machines. Like the JVM, the attack surface will always be huge and security issues nearly endless.

158 points by alpb 7y ago

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on steroids. This architecture allows it to provide a flexible resource footprint (i.e. one based on threads and memory mappings, not fixed guest physical resources) while also lowering the fixed costs of virtualization. However, this comes at the price of reduced application compatibility and higher per-system call overhead. From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). So, you have pretty much complete control over what your apps can do. This for me, is sort of the key takeway from the blog post too: "because we use gVisor to increase the security of Google's own internal workloads, it continuously benefits from our expertise and experience running containers at scale in a security-first environment". So, Google's using something like this internally too for their own workloads, which should be a pretty good sign this works in real life. [1] https://gvisor.dev/docs/architecture_guide/ https://gvisor.dev/docs/architecture_guide/ [2] https://github.com/google/gvisor https://github.com/google/gvisor [3] https://gvisor.dev/ https://gvisor.dev/

roryrjb 7y ago
Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.
- WestCoastJustin 7y ago
  > re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. [1] https://www.cyberark.com/threat-research-blog/how-i-hacked-play-with-docker-and-remotely-ran-code-on-the-host/ https://www.cyberark.com/threat-research-blog/how-i-hacked-p...
  - lima 7y ago
    > Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module. Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.
    - WestCoastJustin 7y ago
      Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?
      - lima 7y ago
        gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit. "people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling. Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.
thesandlord 7y ago
> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contract#sandbox https://cloud.google.com/run/docs/reference/container-contra...
- WestCoastJustin 7y ago
  Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative
  - spyspy 7y ago
    The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
    - cameronbrown 7y ago
      > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
      - deleted 7y ago
        [deleted]
      - asciimike 7y ago
        Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
prattmic 7y ago
> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy configuration for your application; just put your application inside gVisor and go. These are significant differences over simply using something like seccomp on its own (what the architecture guide calls "Rule-based execution"). Some of this is covered in our security model: https://gvisor.dev/docs/architecture_guide/security/#principles-defense-in-depth https://gvisor.dev/docs/architecture_guide/security/#princip...
- WestCoastJustin 7y ago
  Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too (https://news.ycombinator.com/item?id=16976392 https://news.ycombinator.com/item?id=16976392).
  - bradfitz 7y ago
    App Engine now uses gVisor too: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
- saagarjha 7y ago
  Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?
  - prattmic 7y ago
    For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2]. > how do you implement features that ostensibly require calls to the OS anyways? gVisor's kernel is a user-space program, so it can and does make system calls to the host OS. Some examples: * An application blocks trying to read(2) from a pipe. gVisor ultimately implements blocking by waiting on a Go channel. The Go runtime will ultimately implement this with a futex(2) call to the host OS. * An application reads from a file that is ultimately backed by a file on the host (provided by the Gofer [3]). This will result in a pread(2) system call to the host. The purpose here isn't to avoid the host completely (that's not possible), but to limit exposure to the host. gVisor can implement all the parts of Linux it does on a much smaller subset of host system calls. Anything we don't use is blocked by a second-level seccomp sandbox around the kernel. e.g., the kernel cannot make obscure system calls, or even open files or create sockets on the host (those operations are controlled by an external agent). [1] https://github.com/google/gvisor/tree/master/test/syscalls/linux https://github.com/google/gvisor/tree/master/test/syscalls/l... [2] https://github.com/google/syzkaller https://github.com/google/syzkaller [3] https://gvisor.dev/docs/architecture_guide/overview/ https://gvisor.dev/docs/architecture_guide/overview/
equalunique 7y ago
My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s https://www.youtube.com/watch?v=Ur0hbW_K66s I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM https://www.youtube.com/watch?v=HgtRAbE1nBM
blaisio 7y ago
It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did). This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really just in a big technology cycle. Dynamically typed -> statically typed -> dynamically typed; bare metal -> API wrappers -> VMs -> containers -> API wrappers. Soon we'll probably be back to bare metal.
- ryacko 7y ago
  There are web hosts offer Raspberry PIs, but they tend to be more expensive than VMs. I'm guessing colocation costs are dominant.

conroy 7y ago

For those more familiar with Kubernetes and gVisor, would this allow me to build a CI/CD service that runs untrusted user code?

snug 7y ago
> gVisor provides a virtualized environment in order to sandbox untrusted containers. So yes
raesene9 7y ago
Well like all things in security, that kind of depends :) What gVisor does is provide a smaller attack surface to a containerized process, when compared with a "traditional" Docker container using standard Docker setup (you can, of course harden Docker containers considerably from base, if you are so inclined). However it doesn't affect anything outside of that interface so, for example, if your CI/CD process is running on a network that has other insecure services on them, then gVisor alone won't really help you if malicious code is executed inside a container allowing an attacker to start probing the environment from the perspective of that container.

raesene9 7y ago

This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm really not a fan of the opening line where it references the old trope of "containers don't contain" The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. Have there been vulns that allow for container breakout, sure there have, but every piece of software (including gVisor) has had vulnerabilities in it. What you can say about gVisor is that it likely presents a smaller attack surface in its default configuration than a runc style Docker container. However, of course, there's nothing to stop people tightening up on the defaults and still using runc. As an aside for anyone who thinks container breakouts are trivially easy, you can go to https://contained.af https://contained.af and win yourself some money :)

longtermsec 7y ago
The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. -- not just being contrarian here, actually, the reality is that it might be trivial. and it was demonstratively trivial for a long time (see CVE-2019-5736) As for contained.af -- its not a good indicator, it mostly indicates that the reward doesnt meet the market price for demonstrating an escape from a set of hardened namespaces (which is going to cost more than an escape from "any docker container").
- raesene9 7y ago
  So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I was talking about, all software has vulns, including runc, including gvisor. Stating that "containers don't contain" implies that it's not just a specific bug, but that architecturally the process is flawed (at least IMHO), which I would suggest is at the least an over-simplification. as to contained.af, well if it was indeed "trivial" then surely not a large reward would be required :)

bogomipz 7y ago

I have read that all containers at Google run inside of a VM and indeed that article mentions that gVisor is in use in things like App Engine and their internal workloads. So if containers on GKE were already being spun up inside lightweights VMs what does allowing customer's to select the gVisor runtime offer beyond whatever Google's existing lightweight VM already provides?

thesandlord 7y ago
Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.
- bogomipz 7y ago
  What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1] Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around: https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13 [1] https://queue.acm.org/detail.cfm?id=2898444 https://queue.acm.org/detail.cfm?id=2898444
  - yoshiat 7y ago
    (Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases. [1] https://www.youtube.com/watch?v=TQfc8OlB2sg https://www.youtube.com/watch?v=TQfc8OlB2sg

muricula 7y ago

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/ https://gvisor.dev/docs/architecture_guide/performance/

lifty 7y ago
where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.
- muricula 7y ago
  Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.
  - lifty 7y ago
    Missed that one. The gVisor calls are closer to 40k but the difference is indeed big.
  - amscanne 7y ago
    It's just log, not log log. The chart generates from .csv hosted on the site [1] and the benchmark tools are all open. The ptrace numbers are 20x and the KVM platform is actually lower than the Docker default case (though that doesn't mean everything is faster, as system call time is only one factor). As I note above, I think you're confused about what the KVM platform is -- it's not a VM. [1] https://gvisor.dev/performance/syscall.csv https://gvisor.dev/performance/syscall.csv
danbeaulieu 7y ago

whalesalad 7y ago

Isn't this server-side react rendering? What are we doing? We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something? Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technology with the ergonomics of containers and then we'll be WAY closer to true portability and better security.

lemoncucumber 7y ago
One reason I can see is the same reason that linuxkit [1] is a really interesting way of putting together Linux OS images: the container ecosystem has produced some tools that are really useful for building and packaging up Linux userspaces, and being able to reuse those tools in other ways is valuable. With linuxkit you give up some of the niceties of image layer caching, but with an approach like this you get the best of both worlds -- the isolation of VMs but the tooling and usability of containers. [1] https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit
amscanne 7y ago
(I'm a co-author of the blog post) Many functions of the kernel are still effectively shared: memory management (e.g. reclaim, swap), thread scheduling, etc. The application is simply limited in its ability to interact with the shared kernel, and functionality related to system APIs is isolated. Arguably I think this is closer to the ergonomics of containers, but with compatibility and performance trade-offs.
dastbe 7y ago
(I work at AWS) Have you seen https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... ? The team here is working to make firecracker as seamless as possible for running containerized applications in a microvm.

ronsor 7y ago

At this point, why not just use a virtual machine? We've come full circle!

jacques_chester 7y ago
Mostly the performance characteristics. A virtual machine presenting as a machine needs an operating system to be useful. Most operating systems have long-engrained assumptions about the nature of the world, such as: "There is a time when I go from power-off to power-on, and it is rare, so I may perform expensive operations then to amortise their cost over running time". or "While running, time does not skip and hardware does not change". The practical upshot being that the OS needs to be booted from scratch in a number of scenarios. But it's not the OS that provides value. It's a means to an end, and that end is to run software. Most software written to run on OSes also have engrained assumptions, such as "I will come to be launched on a fully-booted system". Containers move the virtualisation up from hardware to the OS API surface. Because the cost of booting is now amortised over all containers running on the system, the original assumptions of both OS designers and software designers become, approximately, true again. So you're right, we came full circle, but not to a point that means "use fully-dressed VMs again".

andrewstuart 7y ago

Sandboxed containers with kernels - so what's the difference now between this and fully isolated virtual machine? Another approach might be to make virtual machine technology more like containers. Then the two shall meet.

edoo 7y ago
I didn't dig into the implementation details but the term para-hvm came to mind, not quite para virtual but not quite full hvm. Perhaps if security is a real issue then HVM is the only real choice.

metta2uall 7y ago

I quite like this defense-in-depth approach, but it's disappointing that it will only be available as part of the probably expensive GKE Advanced. I would have thought safety features should be standard..

dilyevsky 7y ago
I think either way control plane is free now?
- metta2uall 7y ago
  Well gVisor doesn't use the control plane. It is free, but I wouldn't think it has a high cpu or memory load, and Google would make a lot of profit on the nodes.
  - dilyevsky 7y ago
    I know but they may conceivably just charge fixed fee for enabling that option on the nodepool. > it has a high cpu or memory load, and Google would make a lot of profit on the nodes. They currently solve that problem by having their node VMs melt down at like 50% utilization so you have to run everything with huge padding.

fulafel 7y ago

gVisor is easy to run in on your local dev laptop Docker too. It's a nice alternative to running Docker in a VM, if you prefer a security boundary between random Docker containers you get off Docker Hub and your host machine. After you built or downloaded the gVisor single binary to /usr/local/bin/ or wherever, just put a snippet provided in the gVisor README in the Docker settings file ("runtime": {...}), and bob's your uncle.

nullwasamistake 7y ago

The only advantage I see to containers over VM is RAM sharing. Beyond that, hardware VM's are better performing and much more secure. gVisor is just another flavor of containers that replaces kernel interfaces with a Go shim layer to reduce the attack surface in return for worse performance. If somebody could hack ram sharing/overcommit into traditional VM's all this container nonsense could be dispensed with. Containers are a virtualization layer just like the old days when we used the JVM to run "safe" applets on client machines. Like the JVM, the attack surface will always be huge and security issues nearly endless.