I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workaroun

I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds (https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md https://github.com/firecracker-microvm/firecracker/blob/main...) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... IPSec/IKE VPN? May not work (iirc this requires some specific paravirt features? I might be misremembering). Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copied from an I/O thread of an emulated network device to a host TAP device, before it makes it to the real device; my guess is network latency and maximum PPS were not a priority. - The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock. - All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure. - Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck. - The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising the guest driver could be a serious issue. I don't know if Kata does this differently. - Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines. - This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main/docs/kernel-policy.md https://github.com/firecracker-microvm/firecracker/blob/main... They emphasize that they want your host and guest to run "a supported kernel", even if it's possible to run different ones. To me that says that there's the potential for Firecracker-specific bugs in newer (or older?) guest kernels. From that page: "Firecracker represents a component in a larger stack, one in which it is tightly coupled with the guest and host kernels on which it is run." I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.