8 ms·

I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exac

I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't even live migrate a firecracker VM right now. So yes, VMs are great and will ultimately resurge to the detriment of 'everything is a container' serverless orchestration, but Firecracker in its current form won't cut it. A gulf exists between giant cloud operators and their lucrative 'serverless' (only) model and everyone else. Everyone else must avoid the insurmountable costs associated with abandoning the immense value that stateful VMs provide and the minute something emerges that delivers a seamless VM orchestration system it will skyrocket.

tptacek 4y ago
Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
- 0xbadcafebee 4y ago
  From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".
  - tptacek 4y ago
    Can you be specific about what those hardware things are, and why they matter for serverside workloads? Yes: the premise of Firecracker is that the hypervisor only supports the virtio devices, and doesn't emulate real hardware. Where does that end up being problematic? I'm not sure what you mean by your "CPU and memory" being "limited", either.
    - 0xbadcafebee 4y ago
      I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds (https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md https://github.com/firecracker-microvm/firecracker/blob/main...) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... IPSec/IKE VPN? May not work (iirc this requires some specific paravirt features? I might be misremembering). Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copied from an I/O thread of an emulated network device to a host TAP device, before it makes it to the real device; my guess is network latency and maximum PPS were not a priority. - The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock. - All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure. - Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck. - The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising the guest driver could be a serious issue. I don't know if Kata does this differently. - Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines. - This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main/docs/kernel-policy.md https://github.com/firecracker-microvm/firecracker/blob/main... They emphasize that they want your host and guest to run "a supported kernel", even if it's possible to run different ones. To me that says that there's the potential for Firecracker-specific bugs in newer (or older?) guest kernels. From that page: "Firecracker represents a component in a larger stack, one in which it is tightly coupled with the guest and host kernels on which it is run." I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.
mbreese 4y ago
Why would you want to migrate a firecracker vm? They are supposed to be for small, short lived processes. Or in this case, Docker containers (which are likely to be controlled by some kind of orchestration). If you want to be able to migrate a running VM, do you still really need it to start super fast? It’s going to be a long lived service (otherwise you wouldn’t migrate it), so a longer spin up time should be acceptable. If that’s the case, then you can use a different VM tech that supports migration (KVM/Xen/etc).
- 0xbadcafebee 4y ago
  I think that is GP's point: Firecracker shouldn't be considered a general-purpose solution for more secure containerized workloads.
  - tptacek 4y ago
    Can you try to make that a little more specific? What does "general-purpose" mean here? "Secure containerized workloads" tends to imply serverside. It's hard to think of things --- other than GPU access --- that Firecracker takes away here.
    - topspin 4y ago
      > It's hard to think of things No, it isn't. Investigate the K8S world and observe the backflips and somersaults being performed there to deal with networking and storage. Much of that is a consequence of people trying make containers do things that should be run in VMs. A full featured Linux kernel and certain key user space tools can obviate much of that nonsense with ease, deliver better performance, use less resources and be vastly less complex.
      - tptacek 4y ago
        If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.

tptacek 4y ago

Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.

0xbadcafebee 4y ago
From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".
- tptacek 4y ago
  Can you be specific about what those hardware things are, and why they matter for serverside workloads? Yes: the premise of Firecracker is that the hypervisor only supports the virtio devices, and doesn't emulate real hardware. Where does that end up being problematic? I'm not sure what you mean by your "CPU and memory" being "limited", either.
  - 0xbadcafebee 4y ago
    I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds (https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md https://github.com/firecracker-microvm/firecracker/blob/main...) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... IPSec/IKE VPN? May not work (iirc this requires some specific paravirt features? I might be misremembering). Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copied from an I/O thread of an emulated network device to a host TAP device, before it makes it to the real device; my guess is network latency and maximum PPS were not a priority. - The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock. - All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure. - Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck. - The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising the guest driver could be a serious issue. I don't know if Kata does this differently. - Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines. - This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main/docs/kernel-policy.md https://github.com/firecracker-microvm/firecracker/blob/main... They emphasize that they want your host and guest to run "a supported kernel", even if it's possible to run different ones. To me that says that there's the potential for Firecracker-specific bugs in newer (or older?) guest kernels. From that page: "Firecracker represents a component in a larger stack, one in which it is tightly coupled with the guest and host kernels on which it is run." I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.

mbreese 4y ago

Why would you want to migrate a firecracker vm? They are supposed to be for small, short lived processes. Or in this case, Docker containers (which are likely to be controlled by some kind of orchestration). If you want to be able to migrate a running VM, do you still really need it to start super fast? It’s going to be a long lived service (otherwise you wouldn’t migrate it), so a longer spin up time should be acceptable. If that’s the case, then you can use a different VM tech that supports migration (KVM/Xen/etc).

0xbadcafebee 4y ago
I think that is GP's point: Firecracker shouldn't be considered a general-purpose solution for more secure containerized workloads.
- tptacek 4y ago
  Can you try to make that a little more specific? What does "general-purpose" mean here? "Secure containerized workloads" tends to imply serverside. It's hard to think of things --- other than GPU access --- that Firecracker takes away here.
  - topspin 4y ago
    > It's hard to think of things No, it isn't. Investigate the K8S world and observe the backflips and somersaults being performed there to deal with networking and storage. Much of that is a consequence of people trying make containers do things that should be run in VMs. A full featured Linux kernel and certain key user space tools can obviate much of that nonsense with ease, deliver better performance, use less resources and be vastly less complex.
    - tptacek 4y ago
      If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.