4 ms·

I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight

by rwmj 3y ago

I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes). The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers. Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.

cogman10 3y ago
> but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.
- tracker1 3y ago
  I think in the longer run, WASM might displace a lot of both in practical terms.
  - ehutch79 3y ago
    Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.
    - bombela 3y ago
      wasm targets the wasm runtime Virtual Machine (ie: a JavaScript VM). Offering fine grained isolation compared to virtualizing the whole operating system. edit: don't shoot the messenger. I was merely highlighting the main difference between native and webasm in the context of the discussion.
      - meepmorp 3y ago
        You could say the same thing for anything that targets JVM or CLR, and they're far more mature than any JS runtime.
        outworlder 3y ago
        Or even Lua, which is trivial to sandbox.
    - skybrian 3y ago
      Something still compiles a WASM binary for the local system. Possibly, being able to optimize the WASM without recompiling it from source might be a win? Not needing separate binaries for ARM and x86 is nice, so it should run on a Mac more easily. Also, it runs on an edge server or in a browser, even on a phone, if you care about that. I don’t think it will replace Docker files since they let you package up such a wide variety of existing server software and WASM is more limited. But if your software does compile to WASM then maybe you don’t care about that. I think of WASM more like a plugin format, but I expect there will be a lot of engineering effort put into optimizing it, like happened with V8 for JavaScript. Not all web standards win, but betting against one that’s well-established and has a lot of support seems like a mistake.
  - wongarsu 3y ago
    For serverless (as in AWS-lambda-like) I agree. In that usecase WASM provides a better security barrier than containers, with faster cold-start time (which is really important for the scaling promise of these services). For the stuff people run on their Kubernetes clusters I have more mixed expectations. Containers are more universal, but I can totally see a microservice architecture running as a lot of WASM runtimes with a handful of containers.
    - cogman10 3y ago
      Particularly in the case of Amazon lambdas, those are running in VMS already (firecracker). Why wouldn't you skip the VM and use a static binary from rust instead?
hodgesrm 3y ago
> The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Conversely the problem with containers is that memory allocation including the OS page cache is not guaranteed. That's bad for a lot of applications, especially databases. It seems Docker has some support for shared page cache but it's not in the Kubernetes pod spec as far as I can see. [0] You would probably need some kind of annotations and a specialized controller to make this work. [0] https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916
- dharmab 3y ago
  In Kubernetes 1.25+ page cache usage accounting is improved thanks to use of cgroupsv2. https://kubernetes.io/docs/concepts/architecture/cgroups/ https://kubernetes.io/docs/concepts/architecture/cgroups/
mochomocha 3y ago
Slightly off topic, but regarding the larger memory footprint of Kata containers, what is your opinion on KSM effectiveness in general for VMs?
- rwmj 3y ago
  We had a lot of reports of ksm/ksmtuned consuming a lot of CPU and not making a lot of difference. I think it works well for certain workloads, and can be quite pessimal for others. There are also security concerns because you can leak information about (eg) what glibc is being used by another tenant using timing attacks. So you'd probably want to turn it off if multiple tenants can be using a single node.
nimish 3y ago
Kata containers support memory ballooning like most modern VMs: https://en.wikipedia.org/wiki/Memory_ballooning https://en.wikipedia.org/wiki/Memory_ballooning so a fixed allocation isn't needed, reducing over provisioning https://github.com/kata-containers/kata-containers/blob/d50f3888af2c1dee7afeb7a9b77956ce5434eac8/docs/how-to/how-to-use-virtio-mem-with-kata.md https://github.com/kata-containers/kata-containers/blob/d50f... uses virtio-mem
- rwmj 3y ago
  This isn't a substitute (nor is virtio-mem, the modern equivalent). The problem is the application running in userspace inside the guest cannot request more memory when, for example, it does a mmap or sbrk.
  - deleted 3y ago
    [deleted]
  - nimish 3y ago
    Interesting, that's a very annoying constraint then
    - linlinjin 3y ago
      i couldn't understand the comment that "the application running in user space cannot request more memory" - can someone explain whats the point of memorybalooning anywhere if an application cannot signal when the system should actually provision physical memory from the 'baloon'
      - rwmj 3y ago
        There isn't a point, that's the problem.
        Borealid 3y ago
        The systems administrator can use ballooning to give more memory to a VM before launching a new application. This avoids the need to shut down the VM to give it a new role. There is still a benefit to ballooning support even if it's not exposed to userspace within the VM, because VMs aren't always used purely to host a single infinitely-long-lived application without outside intervention.
sitkack 3y ago
Linux supports hot attach ram or the VMM could support memory ballooning. VMs don’t necessarily need to all be backed by physical RAM.

by rwmj 3y ago

cogman10 3y ago

> but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.

tracker1 3y ago
I think in the longer run, WASM might displace a lot of both in practical terms.
- ehutch79 3y ago
  Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.
  - bombela 3y ago
    wasm targets the wasm runtime Virtual Machine (ie: a JavaScript VM). Offering fine grained isolation compared to virtualizing the whole operating system. edit: don't shoot the messenger. I was merely highlighting the main difference between native and webasm in the context of the discussion.
    - meepmorp 3y ago
      You could say the same thing for anything that targets JVM or CLR, and they're far more mature than any JS runtime.
      - outworlder 3y ago
        Or even Lua, which is trivial to sandbox.
  - skybrian 3y ago
    Something still compiles a WASM binary for the local system. Possibly, being able to optimize the WASM without recompiling it from source might be a win? Not needing separate binaries for ARM and x86 is nice, so it should run on a Mac more easily. Also, it runs on an edge server or in a browser, even on a phone, if you care about that. I don’t think it will replace Docker files since they let you package up such a wide variety of existing server software and WASM is more limited. But if your software does compile to WASM then maybe you don’t care about that. I think of WASM more like a plugin format, but I expect there will be a lot of engineering effort put into optimizing it, like happened with V8 for JavaScript. Not all web standards win, but betting against one that’s well-established and has a lot of support seems like a mistake.
- wongarsu 3y ago
  For serverless (as in AWS-lambda-like) I agree. In that usecase WASM provides a better security barrier than containers, with faster cold-start time (which is really important for the scaling promise of these services). For the stuff people run on their Kubernetes clusters I have more mixed expectations. Containers are more universal, but I can totally see a microservice architecture running as a lot of WASM runtimes with a handful of containers.
  - cogman10 3y ago
    Particularly in the case of Amazon lambdas, those are running in VMS already (firecracker). Why wouldn't you skip the VM and use a static binary from rust instead?

hodgesrm 3y ago

> The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Conversely the problem with containers is that memory allocation including the OS page cache is not guaranteed. That's bad for a lot of applications, especially databases. It seems Docker has some support for shared page cache but it's not in the Kubernetes pod spec as far as I can see. [0] You would probably need some kind of annotations and a specialized controller to make this work. [0] https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916

dharmab 3y ago
In Kubernetes 1.25+ page cache usage accounting is improved thanks to use of cgroupsv2. https://kubernetes.io/docs/concepts/architecture/cgroups/ https://kubernetes.io/docs/concepts/architecture/cgroups/

mochomocha 3y ago

Slightly off topic, but regarding the larger memory footprint of Kata containers, what is your opinion on KSM effectiveness in general for VMs?

rwmj 3y ago
We had a lot of reports of ksm/ksmtuned consuming a lot of CPU and not making a lot of difference. I think it works well for certain workloads, and can be quite pessimal for others. There are also security concerns because you can leak information about (eg) what glibc is being used by another tenant using timing attacks. So you'd probably want to turn it off if multiple tenants can be using a single node.

nimish 3y ago

Kata containers support memory ballooning like most modern VMs: https://en.wikipedia.org/wiki/Memory_ballooning https://en.wikipedia.org/wiki/Memory_ballooning so a fixed allocation isn't needed, reducing over provisioning https://github.com/kata-containers/kata-containers/blob/d50f3888af2c1dee7afeb7a9b77956ce5434eac8/docs/how-to/how-to-use-virtio-mem-with-kata.md https://github.com/kata-containers/kata-containers/blob/d50f... uses virtio-mem

rwmj 3y ago
This isn't a substitute (nor is virtio-mem, the modern equivalent). The problem is the application running in userspace inside the guest cannot request more memory when, for example, it does a mmap or sbrk.
- deleted 3y ago
  [deleted]
- nimish 3y ago
  Interesting, that's a very annoying constraint then
  - linlinjin 3y ago
    i couldn't understand the comment that "the application running in user space cannot request more memory" - can someone explain whats the point of memorybalooning anywhere if an application cannot signal when the system should actually provision physical memory from the 'baloon'
    - rwmj 3y ago
      There isn't a point, that's the problem.
      - Borealid 3y ago
        The systems administrator can use ballooning to give more memory to a VM before launching a new application. This avoids the need to shut down the VM to give it a new role. There is still a benefit to ballooning support even if it's not exposed to userspace within the VM, because VMs aren't always used purely to host a single infinitely-long-lived application without outside intervention.

sitkack 3y ago

Linux supports hot attach ram or the VMM could support memory ballooning. VMs don’t necessarily need to all be backed by physical RAM.