8 ms·

Kata Containers – The speed of containers, the security of VMs

208 points by bharatkhatri14 9y ago

reacharavindh 9y ago
Impressive backing by the big name companies. The idea of treating containers as secure and isolated as VMs is enticing for non-ephemeral services. Are these strictly tuned to exploit intel Hardware features or would they consider supporting the equivalent features in say AMD? On the other hand, isn't this the realm of mainline distributions like RHEL, Debian and the like? To support such isolation facilities. I always thought clear Linux was a Intel playground for proof-of-concept which will eventually be up streamed to major Linux distributions.Is it not true? I guess my question is why a separate project like this, instead of RedHat Enterprise Containers or Debian containers?
- FooBarWidget 9y ago
  > The idea of treating containers as secure and isolated as VMs is enticing for non-ephemeral services Are you saying that security and isolation is not enticing for ephemeral services? I know that an ephemeral container is reset after a restart but I think that it's a bit naive to think that that is a good enough replacement for true isolation.
  - reacharavindh 9y ago
    > Are you saying that security and isolation is not enticing for ephemeral services? Didn't mean to imply the reverse logic of my statement. I believe Linux Containers (and hence Docker) depend only on Kernel namespaces to provide isolation. In my admittedly naive eyes, they were not good enough/mature to replace my KVM VMs yet. Too much to trade off for little convenience/performance. However, if Linux containers matured up and offered the same isolation facilities that something like KVM does, then I can think about switching to them in future, and enjoy the performance boost. >I know that an ephemeral container is reset after a restart but I think that it's a bit naive to think that that is a good enough replacement for true isolation. If I'm looking to run an application for which I care about solid isolation of resources, I'd spend my time running it as VM. But, if I'm running a one-time script that chews some data and I don't care much about it bothering other workloads in the system or other workloads bothering it, then I'd fall back on the isolation facilities offered by namespaces by using Containers. Nothing wrong with that. Security view on these is another argument. If I can't afford the application escalating it's view and looking into other workloads in that system, I just wouldn't run them in Containers today.
    - devonkim 9y ago
      Sounds like you are looking for something closer to LXD or perhaps Rkt.
      - reacharavindh 9y ago
        But, AFAIK, LXD and RKt are similar to Docker as a container runtime though. They all share the host kernel, and if one container is hosed/tainted, your host kernel becomes the attack vector. If I read correctly, hypercontainer/kata containers lets you bring your own kernel for your containers and isolates it from the host using intel hardware features(same ones that KVM leverages). That's where it gets interesting to me.
        bonzini 9y ago
        Kata Containers uses KVM; QEMU, which is the userspace KVM client, is configured so that it looks like you are running on a container. However, what you get is indeed a virtual machine. It is simply impossible for "real" containers to provide the same isolation as virtual machine, simply because the attack surface is that of the shared kernel; a hypervisor presents a much more constrained interface to a VM than the full kernel, even if you add QEMU to the mix.
        pstuart 9y ago
        Silly question: if the KVM is using para-virtualized drivers and there is a vulnerability in same, then the host kernel would still be vulnerable?
        bonzini 9y ago
        Many of KVM's paravirtualized drivers run in QEMU, and in turn that is usually running heavily confined, for example using SELinux. So it's true that, as in the famous Theo de Raadt rant, virtualization overall adds to the attack surface compared to containers. But it would also be stupid to ignore that it also introduces very important bottlenecks: to get to the hypervisor, you have to break KVM which is only ~60000 lines of code; getting remote code execution in QEMU might be easier, but then you also have to break the host kernel from a process that has access to almost nothing on the system. There is also vhost, which is implementing PV drivers in the host kernel. This however is also a small amount of code, and it is generally used only for networking and AF_VSOCK sockets.
- odiroot 9y ago
  Interesting that nearly half of the backers are Chinese companies.
  - supermatt 9y ago
    Not when you consider that half of all companies are Chinese companies. For comparison: USA: ~30m China: ~80m
- perlgeek 9y ago
  > On the other hand, isn't this the realm of mainline distributions like RHEL, Debian and the like? At least Debian doesn't develop isolation solutions on its own; it tends to package software that's already out there. And if it's popular enough, it might be integrated fairly tightly into the distribution.
- hacknat 9y ago
  At the Kubecon demo yesterday they cracked a joke about promising to support more architectures. They seem to sincerely want to, which is why they donated the project to OpenStack. Honestly it seems really cool, the spin up time is damn impressive.
- mugsie 9y ago
  Separate projects like this is how a lot of these "RHEL Enterprise $FOO" are actually made. RedHat / Suse / Ubuntu / $Vendor take the upstream project, tidy it a bit, package it, get it integrated in their ecosystem, and add an easy installer. Having it in a vendor neutral foundation means that all the vendors can colaborate, and not have one group with a massive advantage or complete control over a roadmap.
  - bonzini 9y ago
    There are hundreds of engineers working on RHEL (disclaimer, that includes me), so it's not as simple as you put it...
    - mugsie 9y ago
      No, it is not - I over simplified the process a lot. (/me used to work in a vendor, and in a different large Linux distro :) ) I didn't mean to undermine the work that goes into turning a project like this, OpenStack, Kubernetes, Cloud Foundry etc into a real product that users can download and install on random hardware in random configurations, and get a working system - it is a ton of work, and is massively important for getting actual users to install what are very complex distributed systems.
- peterwwillis 9y ago
  > Impressive backing by the big name companies. From someone who works at big name companies: this should not impress anyone. Big companies love slapping their name on things that give them "innovation" credibility. It's like Pepsi sponsoring the X Games. > why a separate project like this This is an OpenStack project, so it's not vendor-specific. It's also supposed to be a new "standard container", which I highly doubt will happen because they're just slapping together two other projects.
tripue 9y ago
Another alternative is using hyper container
- tpetry 9y ago
  This is a combined work of the people behind Intel Clear containers and Hyper containers.
mnd999 9y ago
The British Indian Ocean territory really is becoming a tech hub.
- oblio 9y ago
  I'm not sure I get the connection. Also: https://en.m.wikipedia.org/wiki/Depopulation_of_Chagossians_from_the_Chagos_Archipelago https://en.m.wikipedia.org/wiki/Depopulation_of_Chagossians_...
  - CapacitorSet 9y ago
    >I'm not sure I get the connection. .io is the TLD for the British Indian Ocean Territory, technically speaking.
- dmytrish 9y ago
  So there is no country called "Input/Output", what a bummer.
- techiferous 9y ago
  That place has some sad history: https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_from_the_Chagos_Archipelago https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_fr...
perlgeek 9y ago
One thing that isn't mentioned on front page at least is the management aspect. Docker became popular because it was pretty easy to use, and to publish and reuse existing containers. Whatever competes with it only stands a chance if it can either reuse the existing container ecosystem, or offer something roughly as good.
- kuschku 9y ago
  It doesn’t have to replace Docker – just becoming a better container engine as backend for Kubernetes will be very useful.
- jchw 9y ago
  If it's using Hyper runV, my guess is that their intent is to be compatible with OCI and Docker.
- paulfurtado 9y ago
  Sat through the talk at kubecon yesterday - an important goal of theirs is to not compete with the docker. They said it was compatible with docker, containerd, and cri-o. I believe with docker, it sits at the runc level, so to the end user, you're using docker in the standard fashion, but the underlying isolation mechanism is different. They also said it can be chosen per container so different containers on the same host can use different isolation mechanisms
  - perlgeek 9y ago
    That makes a lot of sense, and probably the road that makes adoption easiest. Thanks!
chungy 9y ago
It's kind of interesting that it's only in the Linux world that containers cannot be thought of as isolated or secure. Seeing it from a jails and zones perspective, rather sad, actually :)
- oblio 9y ago
  Does Windows have anything like this?
  - kuschku 9y ago
    Yes – HyperV containers (which Kata is actually inspired by) are much more secure than Linux’ namespaces.
    - tilpner 9y ago
      > Kata Containers combines technology from Intel® Clear Containers and Hyper runV but I can't find a mention of Hyper-V anywhere (which doesn't mean there was no inspiration). Maybe you confused Hyper runv and Hyper-V here (the naming certainly doesn't help)?
      - cbzbc 9y ago
        runV is a oci compatible drop in replacement for runC that can execute containers on a number of backend virtualisation environments, including Hyper-V and KVM
        jchw 9y ago
        That's coincidence, though. runV wasn't inspired by Hyper-V.
      - kuschku 9y ago
        I might have just been confused due to the naming, but, as far as I can see, they’re using the exact same underlying technology, based on AMD’s and Intel’s virtualization extensions, to replace the sandboxing that is currently handled by kernel namespaces, jails, or HyperV containers (and, in some of these implementations, already uses this technology)
jeshwanth 9y ago
Whats the difference between unikernels and kata containers?
- jchw 9y ago
  Different approaches to isolation. A kata container is using Clear Linux to load a feature-complete Linux kernel into tiny VMs (disclaimer: I do not know exactly how it's different from any other VM,) a unikernel is a small bare-metal "library" that gives you minimal OS-like functions to put in a hypervisor to run your application. Unikernels are still more minimal, I'd guess.
  - ams6110 9y ago
    Here's a recent paper about the unikernel approach http://cnp.neclab.eu/projects/lightvm/lightvm.pdf http://cnp.neclab.eu/projects/lightvm/lightvm.pdf
jeremyjh 9y ago
They don't seem to have written any code yet. [1] So what we have at this point is a marketing website about their ambition and goals? [1]https://github.com/kata-containers/runtimes https://github.com/kata-containers/runtimes
- paulfurtado 9y ago
  The code comes from Intel's Clear Containers and hyper. The interesting bit is that the tech is now part of the openstack foundation, under the name Kata Containers. At Kubecon yesterday, they did a demo, showing a fork bomb taking out a container, but not the host. It actually seems nearly ready to use.
  - redtuesday 9y ago
    Can't you just combat fork bombs with e.g docker run --pids-limit=64
    - paulfurtado 9y ago
      Yes, there are several ways to combat fork bombs (ulimits or pid namespaces). This was purely for the sake of the live demo that required a kernel crash example, there are certainly other ways to combat it.
- tecleandor 9y ago
  It is comprised by many parts that, already, have seen development as: Kata Agent: https://github.com/kata-containers/agent https://github.com/kata-containers/agent Kata Shim: https://github.com/kata-containers/shim https://github.com/kata-containers/shim Kata Proxy: https://github.com/kata-containers/proxy https://github.com/kata-containers/proxy KSM Throttler: https://github.com/kata-containers/ksm-throttler https://github.com/kata-containers/ksm-throttler And some forks to provide for their necessities, I suppose, as: Linux Kernel: https://github.com/kata-containers/linux https://github.com/kata-containers/linux QEMU: https://github.com/kata-containers/qemu https://github.com/kata-containers/qemu
- 9y ago
e_d_e_v 9y ago
How is this better than using rkt with an lkvm stage1[1], which also uses the work done by the Clear Containers team? It looks like Kata packages QEMU as well, which seems a bit overkill. [1]https://coreos.com/rkt/docs/latest/running-kvm-stage1.html https://coreos.com/rkt/docs/latest/running-kvm-stage1.html
- bonzini 9y ago
  > a bit overkill They also said the same about Xen, that a special purpose microkernel was a better choice than Linux as a hypervisor...
  - e_d_e_v 9y ago
    Right, in many cases, small is beautiful! I think that's what contributed so heavily to the massive success of the Xen platform. Is that what you mean?
    - bonzini 9y ago
      Then why does Kata Containers use KVM? Xen was successful because it was innovative, and because it worked around the fact that x86 was not virtualizable at the time. But after ten years of healthy competition, the only reason to prefer Xen to KVM would be things like QubesOS.
acobster 9y ago
> It is designed to be architecture agnostic, run on multiple hypervisors and be compatible with the OCI specification for Docker containers In what sense is this "OCI compatible"? Do they implement the runtime, image format spec, or both? My understanding of containerization and OCI runtimes is that they're fundamentally different from hardware-level virtualization.
- bergwolf 9y ago
  Both. The hardware virtualization related settings are configured out side of OCI spec but the runtime accept OCI spec and plays with it accordingly. As for image format, Kata runs unchanged docker images.

Impressive backing by the big name companies. The idea of treating containers as secure and isolated as VMs is enticing for non-ephemeral services. Are these strictly tuned to exploit intel Hardware features or would they consider supporting the equivalent features in say AMD? On the other hand, isn't this the realm of mainline distributions like RHEL, Debian and the like? To support such isolation facilities. I always thought clear Linux was a Intel playground for proof-of-concept which will eventually be up streamed to major Linux distributions.Is it not true? I guess my question is why a separate project like this, instead of RedHat Enterprise Containers or Debian containers?

FooBarWidget 9y ago
> The idea of treating containers as secure and isolated as VMs is enticing for non-ephemeral services Are you saying that security and isolation is not enticing for ephemeral services? I know that an ephemeral container is reset after a restart but I think that it's a bit naive to think that that is a good enough replacement for true isolation.
- reacharavindh 9y ago
  > Are you saying that security and isolation is not enticing for ephemeral services? Didn't mean to imply the reverse logic of my statement. I believe Linux Containers (and hence Docker) depend only on Kernel namespaces to provide isolation. In my admittedly naive eyes, they were not good enough/mature to replace my KVM VMs yet. Too much to trade off for little convenience/performance. However, if Linux containers matured up and offered the same isolation facilities that something like KVM does, then I can think about switching to them in future, and enjoy the performance boost. >I know that an ephemeral container is reset after a restart but I think that it's a bit naive to think that that is a good enough replacement for true isolation. If I'm looking to run an application for which I care about solid isolation of resources, I'd spend my time running it as VM. But, if I'm running a one-time script that chews some data and I don't care much about it bothering other workloads in the system or other workloads bothering it, then I'd fall back on the isolation facilities offered by namespaces by using Containers. Nothing wrong with that. Security view on these is another argument. If I can't afford the application escalating it's view and looking into other workloads in that system, I just wouldn't run them in Containers today.
  - devonkim 9y ago
    Sounds like you are looking for something closer to LXD or perhaps Rkt.
    - reacharavindh 9y ago
      But, AFAIK, LXD and RKt are similar to Docker as a container runtime though. They all share the host kernel, and if one container is hosed/tainted, your host kernel becomes the attack vector. If I read correctly, hypercontainer/kata containers lets you bring your own kernel for your containers and isolates it from the host using intel hardware features(same ones that KVM leverages). That's where it gets interesting to me.
      - bonzini 9y ago
        Kata Containers uses KVM; QEMU, which is the userspace KVM client, is configured so that it looks like you are running on a container. However, what you get is indeed a virtual machine. It is simply impossible for "real" containers to provide the same isolation as virtual machine, simply because the attack surface is that of the shared kernel; a hypervisor presents a much more constrained interface to a VM than the full kernel, even if you add QEMU to the mix.
        pstuart 9y ago
        Silly question: if the KVM is using para-virtualized drivers and there is a vulnerability in same, then the host kernel would still be vulnerable?
        bonzini 9y ago
        Many of KVM's paravirtualized drivers run in QEMU, and in turn that is usually running heavily confined, for example using SELinux. So it's true that, as in the famous Theo de Raadt rant, virtualization overall adds to the attack surface compared to containers. But it would also be stupid to ignore that it also introduces very important bottlenecks: to get to the hypervisor, you have to break KVM which is only ~60000 lines of code; getting remote code execution in QEMU might be easier, but then you also have to break the host kernel from a process that has access to almost nothing on the system. There is also vhost, which is implementing PV drivers in the host kernel. This however is also a small amount of code, and it is generally used only for networking and AF_VSOCK sockets.
odiroot 9y ago
Interesting that nearly half of the backers are Chinese companies.
- supermatt 9y ago
  Not when you consider that half of all companies are Chinese companies. For comparison: USA: ~30m China: ~80m
perlgeek 9y ago
> On the other hand, isn't this the realm of mainline distributions like RHEL, Debian and the like? At least Debian doesn't develop isolation solutions on its own; it tends to package software that's already out there. And if it's popular enough, it might be integrated fairly tightly into the distribution.
hacknat 9y ago
At the Kubecon demo yesterday they cracked a joke about promising to support more architectures. They seem to sincerely want to, which is why they donated the project to OpenStack. Honestly it seems really cool, the spin up time is damn impressive.
mugsie 9y ago
Separate projects like this is how a lot of these "RHEL Enterprise $FOO" are actually made. RedHat / Suse / Ubuntu / $Vendor take the upstream project, tidy it a bit, package it, get it integrated in their ecosystem, and add an easy installer. Having it in a vendor neutral foundation means that all the vendors can colaborate, and not have one group with a massive advantage or complete control over a roadmap.
- bonzini 9y ago
  There are hundreds of engineers working on RHEL (disclaimer, that includes me), so it's not as simple as you put it...
  - mugsie 9y ago
    No, it is not - I over simplified the process a lot. (/me used to work in a vendor, and in a different large Linux distro :) ) I didn't mean to undermine the work that goes into turning a project like this, OpenStack, Kubernetes, Cloud Foundry etc into a real product that users can download and install on random hardware in random configurations, and get a working system - it is a ton of work, and is massively important for getting actual users to install what are very complex distributed systems.
peterwwillis 9y ago
> Impressive backing by the big name companies. From someone who works at big name companies: this should not impress anyone. Big companies love slapping their name on things that give them "innovation" credibility. It's like Pepsi sponsoring the X Games. > why a separate project like this This is an OpenStack project, so it's not vendor-specific. It's also supposed to be a new "standard container", which I highly doubt will happen because they're just slapping together two other projects.

tripue 9y ago

Another alternative is using hyper container

tpetry 9y ago
This is a combined work of the people behind Intel Clear containers and Hyper containers.

mnd999 9y ago

The British Indian Ocean territory really is becoming a tech hub.

oblio 9y ago
I'm not sure I get the connection. Also: https://en.m.wikipedia.org/wiki/Depopulation_of_Chagossians_from_the_Chagos_Archipelago https://en.m.wikipedia.org/wiki/Depopulation_of_Chagossians_...
- CapacitorSet 9y ago
  >I'm not sure I get the connection. .io is the TLD for the British Indian Ocean Territory, technically speaking.
dmytrish 9y ago
So there is no country called "Input/Output", what a bummer.
techiferous 9y ago
That place has some sad history: https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_from_the_Chagos_Archipelago https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_fr...

perlgeek 9y ago

One thing that isn't mentioned on front page at least is the management aspect. Docker became popular because it was pretty easy to use, and to publish and reuse existing containers. Whatever competes with it only stands a chance if it can either reuse the existing container ecosystem, or offer something roughly as good.

kuschku 9y ago
It doesn’t have to replace Docker – just becoming a better container engine as backend for Kubernetes will be very useful.
jchw 9y ago
If it's using Hyper runV, my guess is that their intent is to be compatible with OCI and Docker.
paulfurtado 9y ago
Sat through the talk at kubecon yesterday - an important goal of theirs is to not compete with the docker. They said it was compatible with docker, containerd, and cri-o. I believe with docker, it sits at the runc level, so to the end user, you're using docker in the standard fashion, but the underlying isolation mechanism is different. They also said it can be chosen per container so different containers on the same host can use different isolation mechanisms
- perlgeek 9y ago
  That makes a lot of sense, and probably the road that makes adoption easiest. Thanks!

chungy 9y ago

It's kind of interesting that it's only in the Linux world that containers cannot be thought of as isolated or secure. Seeing it from a jails and zones perspective, rather sad, actually :)

oblio 9y ago
Does Windows have anything like this?
- kuschku 9y ago
  Yes – HyperV containers (which Kata is actually inspired by) are much more secure than Linux’ namespaces.
  - tilpner 9y ago
    > Kata Containers combines technology from Intel® Clear Containers and Hyper runV but I can't find a mention of Hyper-V anywhere (which doesn't mean there was no inspiration). Maybe you confused Hyper runv and Hyper-V here (the naming certainly doesn't help)?
    - cbzbc 9y ago
      runV is a oci compatible drop in replacement for runC that can execute containers on a number of backend virtualisation environments, including Hyper-V and KVM
      - jchw 9y ago
        That's coincidence, though. runV wasn't inspired by Hyper-V.
    - kuschku 9y ago
      I might have just been confused due to the naming, but, as far as I can see, they’re using the exact same underlying technology, based on AMD’s and Intel’s virtualization extensions, to replace the sandboxing that is currently handled by kernel namespaces, jails, or HyperV containers (and, in some of these implementations, already uses this technology)

jeshwanth 9y ago

Whats the difference between unikernels and kata containers?

jchw 9y ago
Different approaches to isolation. A kata container is using Clear Linux to load a feature-complete Linux kernel into tiny VMs (disclaimer: I do not know exactly how it's different from any other VM,) a unikernel is a small bare-metal "library" that gives you minimal OS-like functions to put in a hypervisor to run your application. Unikernels are still more minimal, I'd guess.
- ams6110 9y ago
  Here's a recent paper about the unikernel approach http://cnp.neclab.eu/projects/lightvm/lightvm.pdf http://cnp.neclab.eu/projects/lightvm/lightvm.pdf

jeremyjh 9y ago

They don't seem to have written any code yet. [1] So what we have at this point is a marketing website about their ambition and goals? [1]https://github.com/kata-containers/runtimes https://github.com/kata-containers/runtimes

paulfurtado 9y ago
The code comes from Intel's Clear Containers and hyper. The interesting bit is that the tech is now part of the openstack foundation, under the name Kata Containers. At Kubecon yesterday, they did a demo, showing a fork bomb taking out a container, but not the host. It actually seems nearly ready to use.
- redtuesday 9y ago
  Can't you just combat fork bombs with e.g docker run --pids-limit=64
  - paulfurtado 9y ago
    Yes, there are several ways to combat fork bombs (ulimits or pid namespaces). This was purely for the sake of the live demo that required a kernel crash example, there are certainly other ways to combat it.
tecleandor 9y ago
It is comprised by many parts that, already, have seen development as: Kata Agent: https://github.com/kata-containers/agent https://github.com/kata-containers/agent Kata Shim: https://github.com/kata-containers/shim https://github.com/kata-containers/shim Kata Proxy: https://github.com/kata-containers/proxy https://github.com/kata-containers/proxy KSM Throttler: https://github.com/kata-containers/ksm-throttler https://github.com/kata-containers/ksm-throttler And some forks to provide for their necessities, I suppose, as: Linux Kernel: https://github.com/kata-containers/linux https://github.com/kata-containers/linux QEMU: https://github.com/kata-containers/qemu https://github.com/kata-containers/qemu
9y ago

e_d_e_v 9y ago

How is this better than using rkt with an lkvm stage1[1], which also uses the work done by the Clear Containers team? It looks like Kata packages QEMU as well, which seems a bit overkill. [1]https://coreos.com/rkt/docs/latest/running-kvm-stage1.html https://coreos.com/rkt/docs/latest/running-kvm-stage1.html

bonzini 9y ago
> a bit overkill They also said the same about Xen, that a special purpose microkernel was a better choice than Linux as a hypervisor...
- e_d_e_v 9y ago
  Right, in many cases, small is beautiful! I think that's what contributed so heavily to the massive success of the Xen platform. Is that what you mean?
  - bonzini 9y ago
    Then why does Kata Containers use KVM? Xen was successful because it was innovative, and because it worked around the fact that x86 was not virtualizable at the time. But after ten years of healthy competition, the only reason to prefer Xen to KVM would be things like QubesOS.

acobster 9y ago

> It is designed to be architecture agnostic, run on multiple hypervisors and be compatible with the OCI specification for Docker containers In what sense is this "OCI compatible"? Do they implement the runtime, image format spec, or both? My understanding of containerization and OCI runtimes is that they're fundamentally different from hardware-level virtualization.

bergwolf 9y ago
Both. The hardware virtualization related settings are configured out side of OCI spec but the runtime accept OCI spec and plays with it accordingly. As for image format, Kata runs unchanged docker images.