5 ms·

I don't understand how can they be as fast as regular containers if they run an entire kernel on top of an hypervisor ?

by bionsystem 3y ago

Alifatisk 3y ago
Yeah I'd like to see some numbers on that, like startup time.
- theaiquestion 3y ago
  Depending on your use case there's potentially negligible startup time. On the scale of single digit seconds to less then half a second depending on how much work you put into optimizing it. For some applications this will be too slow (mainly the type where you boot a container per request, although flyio seems to make it work), I think for a _lot_ of applications this wouldn't be noticed. Kata gives you a few different options for what/how you'd like to boot including firecracker. This isn't exclusive to firecracker but if you stay lightweight you can have vm's booting under a half second if you're using slim images. https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... I honestly think for a lot of people, vm's with the convenience/orchestration tools of containers make more sense for a lot of general use cases simply because of the security benefits. The convenience still needs some work though.
  - insanitybit 3y ago
    Unless you're dealing with a multi-tenant situation I'm not super convinced that a VM is worth the effort. It's not the perf, it's the need to make your kernel, root file system, and other infra needed to make it all work. Compare that to a docker container where there's basically 0 additional work that has to be done to be up and running. For most cases I'd be really tempted to work on hardening the docker container than on setting up a VM. Things like Apparmor and seccomp in particular would likely go a very long way.
    - deleted 3y ago
      [deleted]
rwmj 3y ago
I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes). The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers. Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.
- cogman10 3y ago
  > but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.
  - tracker1 3y ago
    I think in the longer run, WASM might displace a lot of both in practical terms.
    - ehutch79 3y ago
      Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.
mugsie 3y ago
there is pretty low overhead if you are opinionated - this is very similar to firecracker (AWS) tooling, so cut down hypervisor with ~ 0 devices, and a cut down guest OS means pretty quick boot times
kobalsky 3y ago
there's probably a big asterisk there. the correct term is probably "fast enough" virtualization adds very overhead, a Windows VM running with a dedicated GPU can get 95% of the host's score on 3dmark. the biggest issue on these cases is IO which can be handled in a few ways.
- lproven 3y ago
  > virtualization adds very overhead Missing word? "Very little"? 5% is enough to turn this year's high-end machine into last year's model. Nested virtualisation is also a thing now, and 5% per layer adds up fast.
insanitybit 3y ago
The short version is that kernels support guest/host relationships natively so that guests can pass operations directly to the host without having to go through an additional system call. Everywhere you do this is attack surface where an attacker in the guest can communicate with privileged facilities, so you want to minimize this where you can. There's usually overhead in the places where the communication requires an additional hop. If you want your host filesystem isolated you're going to need a translation layer and it will be slower. If you're willing to open up your host OS's filesystem, you can basically get ~0 overhead.

I don't understand how can they be as fast as regular containers if they run an entire kernel on top of an hypervisor ?

Alifatisk 3y ago

Yeah I'd like to see some numbers on that, like startup time.

theaiquestion 3y ago
Depending on your use case there's potentially negligible startup time. On the scale of single digit seconds to less then half a second depending on how much work you put into optimizing it. For some applications this will be too slow (mainly the type where you boot a container per request, although flyio seems to make it work), I think for a _lot_ of applications this wouldn't be noticed. Kata gives you a few different options for what/how you'd like to boot including firecracker. This isn't exclusive to firecracker but if you stay lightweight you can have vm's booting under a half second if you're using slim images. https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... I honestly think for a lot of people, vm's with the convenience/orchestration tools of containers make more sense for a lot of general use cases simply because of the security benefits. The convenience still needs some work though.
- insanitybit 3y ago
  Unless you're dealing with a multi-tenant situation I'm not super convinced that a VM is worth the effort. It's not the perf, it's the need to make your kernel, root file system, and other infra needed to make it all work. Compare that to a docker container where there's basically 0 additional work that has to be done to be up and running. For most cases I'd be really tempted to work on hardening the docker container than on setting up a VM. Things like Apparmor and seccomp in particular would likely go a very long way.
  - deleted 3y ago
    [deleted]

rwmj 3y ago

I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes). The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers. Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.

cogman10 3y ago
> but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.
- tracker1 3y ago
  I think in the longer run, WASM might displace a lot of both in practical terms.
  - ehutch79 3y ago
    Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.

mugsie 3y ago

there is pretty low overhead if you are opinionated - this is very similar to firecracker (AWS) tooling, so cut down hypervisor with ~ 0 devices, and a cut down guest OS means pretty quick boot times

kobalsky 3y ago

there's probably a big asterisk there. the correct term is probably "fast enough" virtualization adds very overhead, a Windows VM running with a dedicated GPU can get 95% of the host's score on 3dmark. the biggest issue on these cases is IO which can be handled in a few ways.

lproven 3y ago
> virtualization adds very overhead Missing word? "Very little"? 5% is enough to turn this year's high-end machine into last year's model. Nested virtualisation is also a thing now, and 5% per layer adds up fast.

insanitybit 3y ago

The short version is that kernels support guest/host relationships natively so that guests can pass operations directly to the host without having to go through an additional system call. Everywhere you do this is attack surface where an attacker in the guest can communicate with privileged facilities, so you want to minimize this where you can. There's usually overhead in the places where the communication requires an additional hop. If you want your host filesystem isolated you're going to need a translation layer and it will be slower. If you're willing to open up your host OS's filesystem, you can basically get ~0 overhead.