3 ms·

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

bogomipz 7y ago
What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1] Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around: https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13 [1] https://queue.acm.org/detail.cfm?id=2898444 https://queue.acm.org/detail.cfm?id=2898444
- yoshiat 7y ago
  (Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases. [1] https://www.youtube.com/watch?v=TQfc8OlB2sg https://www.youtube.com/watch?v=TQfc8OlB2sg
  - bogomipz 7y ago
    Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?
    - yoshiat 7y ago
      Yes, exactly. https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#nodes https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
- thesandlord 7y ago
  (I work for GCP) It looks something like this: your container -> Compute Engine VM (GKE Node) -> container -> Borg The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail. >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud." As a GCP customer using GKE, your applications are separated from other GCP customer using VMs. However, if you want to run your OWN untrusted workloads, then in the past you would have to spin up a separate VM for untrusted workload A and a one VM for untrusted workload B. This sucks in terms of resource utilization. It would be better in many cases if you could run workload A and B on the same VM. That's where gVisor comes into play. your untrusted container -> gVisor -> Compute Engine VM (GKE Node) -> container -> Borg I hope this makes sense!
  - bogomipz 7y ago
    Thanks for the explanation, this makes sense yes. >"The container on top of Borg is used for scheduling and management." Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then? https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13
    - yoshiat 7y ago
      That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c...
      - bogomipz 7y ago
        I see. So is "the container on top of Borg is used for scheduling and management" the Borg equivalent of the K8S kubelet then?

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

bogomipz 7y ago

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1] Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around: https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13 [1] https://queue.acm.org/detail.cfm?id=2898444 https://queue.acm.org/detail.cfm?id=2898444

yoshiat 7y ago
(Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases. [1] https://www.youtube.com/watch?v=TQfc8OlB2sg https://www.youtube.com/watch?v=TQfc8OlB2sg
- bogomipz 7y ago
  Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?
  - yoshiat 7y ago
    Yes, exactly. https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#nodes https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
thesandlord 7y ago
(I work for GCP) It looks something like this: your container -> Compute Engine VM (GKE Node) -> container -> Borg The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail. >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud." As a GCP customer using GKE, your applications are separated from other GCP customer using VMs. However, if you want to run your OWN untrusted workloads, then in the past you would have to spin up a separate VM for untrusted workload A and a one VM for untrusted workload B. This sucks in terms of resource utilization. It would be better in many cases if you could run workload A and B on the same VM. That's where gVisor comes into play. your untrusted container -> gVisor -> Compute Engine VM (GKE Node) -> container -> Borg I hope this makes sense!
- bogomipz 7y ago
  Thanks for the explanation, this makes sense yes. >"The container on top of Borg is used for scheduling and management." Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then? https://speakerdeck.com/jbeda/containers-at-scale?slide=13 https://speakerdeck.com/jbeda/containers-at-scale?slide=13
  - yoshiat 7y ago
    That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c...
    - bogomipz 7y ago
      I see. So is "the container on top of Borg is used for scheduling and management" the Borg equivalent of the K8S kubelet then?