5 ms·

I'm very surprised the standard isn't to build a microkernel that emulates Linux userspace (or *NIX userspace) and is tailored towards the subset of virtual har

I'm very surprised the standard isn't to build a microkernel that emulates Linux userspace (or *NIX userspace) and is tailored towards the subset of virtual hardware that Firecracker and QEMU provide. I don't get the impression that implementing a new target for a PL is all that difficult, so if you create a psuedo-OS like WASI/WASM and send PRs to the supported languages you could cut out most of the overhead. The "hardest" part is probably sufficiently emulating Linux userspace accurately: it's a big surface area. That's why I think creating a pseudo-OS target is the best route.

tptacek 3y ago
You're describing gvisor.
- packetlost 3y ago
  No, I'm not. Gvisor is a security layer around Linux containers that emulates and constrains syscalls. It specifically runs on top of a container platform and kernel. What I'm suggesting is a stripped down Linux-like kernel that is really good at running exactly one process. I'm describing a microkernel.
  - tptacek 3y ago
    gvisor emulates a big chunk of the Linux system call interface, and, remarkably, a further large chunk of Linux kernel state. It's architecturally similar to uml (though not as complete; like Firecracker, it's optimized to a specific set of workloads). gvisor is not like a seccomp-bpf process sandbox that just ACLs system calls.
    - packetlost 3y ago
      Ok, I oversimplified a bit. Regardless, I'm suggesting something that still runs in emulated hardware isolation and implements drivers for Firecracker/QEMU's subset of hardware.
      - tptacek 3y ago
        gvisor does emulate some hardware. See, for instance, its network stack. At any rate: why is this better than just using KVM and Firecracker? The big problem with gvisor is that the emulation you're talking about has pretty tough overhead.
        packetlost 3y ago
        Let's you get the best of both gvisor and Firecracker: efficient use of resources (ie. not running a full Linux kernel + scheduler, and most importantly, network stack for every lambda) while getting the isolation that comes from virtualization. You can achieve this in one of 2 ways: make a new kernel and add support for targeting it in the supported languages, or strip the Linux kernel down and reimplement the parts that aren't optimized for you short-lived VM lifecycle (scheduler, network stack, etc.).
        tptacek 3y ago
        Stripping the Linux kernel down is what people do with Firecracker. I'm curious what savings you see in the Linux networking stack. You could compile it out and just rely on vsocks, but now you're breaking everyone's code and you're not winning anything on performance.
        packetlost 3y ago
        Perhaps I'm off base (I'm not an expert in this area), but I recall reading that one of the major challenges with Lambda was the latency that initializing the network stack introduces. Perhaps that's been solved by now, but my naive idea is to have the guest not really run it's own network stack (at least the MAC/IP portion of it) and instead delegate the entire stack (IP and all) to the virtual device, which can be implemented by Firecracker/QEMU/whatever. I guess at that point, the amount of mangling you'd need to do to the kernel probably isn't worth it and you should just use Gvisor... ah oh well. Regardless, I'm still surprised microkernels aren't more popular in this space, but perhaps the losing the ecosystem of Linux libs/applications is a non-starter. Even if the idea wasn't fruitful, the conversation was fun. Thanks for engaging and challenging my bad ideas! Edit: I've also realized I was thinking of Unikernels, not microkernels and I've been calling it the wrong thing all night. *sigh*
  - talideon 3y ago
    That's not what a microkernel is. A microkernel is a kernel that pushes services traditionally included in the kernel, such as networking, out into userspace. The closest things to what you're describing are unikernels and NetBSD rump kernels.
    - packetlost 3y ago
      Yeah, further down I realized I was crossing wires and was thinking of a Unikernel. My bad!
  - reachableceo 3y ago
    I believe the term you are looking for is unikernel, not micro kernel.
    - yencabulator 3y ago
      It's a unikernel really only if you rip out any security boundaries inside the VM and link the kernel into the app. If you still maintain a syscall boundary inside the VM, it's just another kernel.
- yencabulator 3y ago
  Sort of. Let's say gVisor is three things: 1. Mechanism to capture syscalls (systrap) 2. Reimplementation of parts Linux kernel 3. Narrow set of calls to outside using Linux syscalls This thing could be envisioned as 1. Linux syscall ABI as-is, same mechanism[a] 2. Reimplementation of parts Linux kernel 3. virt-io hardware drivers for calls to outside So the middle part of the sandwich could look the same. Also, I think it's worth saying that I think the work in maintaining #2 there is exactly why Google Cloud Run migrated away from gVisor. People just kept asking for more and more kernel features. [a]: Alternatively, #1 could be replaced with unikernel like linking directly with #2. Me personally, I think HTTP/3's move away from TCP could be really interesting for this sort of stuff. The responsibilities of the kernel could be hugely simplified if all you had were UDP/IP directly hardcoded to virtio (no need for routing, address configuration, ARP, etc), no paging etc, and the only filesystems were EROFS & tmpfs. Of course, Cloud Run's move away from gVisor shows that Enterprise clients would hate it.

tptacek 3y ago

You're describing gvisor.

packetlost 3y ago
No, I'm not. Gvisor is a security layer around Linux containers that emulates and constrains syscalls. It specifically runs on top of a container platform and kernel. What I'm suggesting is a stripped down Linux-like kernel that is really good at running exactly one process. I'm describing a microkernel.
- tptacek 3y ago
  gvisor emulates a big chunk of the Linux system call interface, and, remarkably, a further large chunk of Linux kernel state. It's architecturally similar to uml (though not as complete; like Firecracker, it's optimized to a specific set of workloads). gvisor is not like a seccomp-bpf process sandbox that just ACLs system calls.
  - packetlost 3y ago
    Ok, I oversimplified a bit. Regardless, I'm suggesting something that still runs in emulated hardware isolation and implements drivers for Firecracker/QEMU's subset of hardware.
    - tptacek 3y ago
      gvisor does emulate some hardware. See, for instance, its network stack. At any rate: why is this better than just using KVM and Firecracker? The big problem with gvisor is that the emulation you're talking about has pretty tough overhead.
      - packetlost 3y ago
        Let's you get the best of both gvisor and Firecracker: efficient use of resources (ie. not running a full Linux kernel + scheduler, and most importantly, network stack for every lambda) while getting the isolation that comes from virtualization. You can achieve this in one of 2 ways: make a new kernel and add support for targeting it in the supported languages, or strip the Linux kernel down and reimplement the parts that aren't optimized for you short-lived VM lifecycle (scheduler, network stack, etc.).
        tptacek 3y ago
        Stripping the Linux kernel down is what people do with Firecracker. I'm curious what savings you see in the Linux networking stack. You could compile it out and just rely on vsocks, but now you're breaking everyone's code and you're not winning anything on performance.
        packetlost 3y ago
        Perhaps I'm off base (I'm not an expert in this area), but I recall reading that one of the major challenges with Lambda was the latency that initializing the network stack introduces. Perhaps that's been solved by now, but my naive idea is to have the guest not really run it's own network stack (at least the MAC/IP portion of it) and instead delegate the entire stack (IP and all) to the virtual device, which can be implemented by Firecracker/QEMU/whatever. I guess at that point, the amount of mangling you'd need to do to the kernel probably isn't worth it and you should just use Gvisor... ah oh well. Regardless, I'm still surprised microkernels aren't more popular in this space, but perhaps the losing the ecosystem of Linux libs/applications is a non-starter. Even if the idea wasn't fruitful, the conversation was fun. Thanks for engaging and challenging my bad ideas! Edit: I've also realized I was thinking of Unikernels, not microkernels and I've been calling it the wrong thing all night. *sigh*
- talideon 3y ago
  That's not what a microkernel is. A microkernel is a kernel that pushes services traditionally included in the kernel, such as networking, out into userspace. The closest things to what you're describing are unikernels and NetBSD rump kernels.
  - packetlost 3y ago
    Yeah, further down I realized I was crossing wires and was thinking of a Unikernel. My bad!
- reachableceo 3y ago
  I believe the term you are looking for is unikernel, not micro kernel.
  - yencabulator 3y ago
    It's a unikernel really only if you rip out any security boundaries inside the VM and link the kernel into the app. If you still maintain a syscall boundary inside the VM, it's just another kernel.
yencabulator 3y ago
Sort of. Let's say gVisor is three things: 1. Mechanism to capture syscalls (systrap) 2. Reimplementation of parts Linux kernel 3. Narrow set of calls to outside using Linux syscalls This thing could be envisioned as 1. Linux syscall ABI as-is, same mechanism[a] 2. Reimplementation of parts Linux kernel 3. virt-io hardware drivers for calls to outside So the middle part of the sandwich could look the same. Also, I think it's worth saying that I think the work in maintaining #2 there is exactly why Google Cloud Run migrated away from gVisor. People just kept asking for more and more kernel features. [a]: Alternatively, #1 could be replaced with unikernel like linking directly with #2. Me personally, I think HTTP/3's move away from TCP could be really interesting for this sort of stuff. The responsibilities of the kernel could be hugely simplified if all you had were UDP/IP directly hardcoded to virtio (no need for routing, address configuration, ARP, etc), no paging etc, and the only filesystems were EROFS & tmpfs. Of course, Cloud Run's move away from gVisor shows that Enterprise clients would hate it.