4 ms·

No, I'm not. Gvisor is a security layer around Linux containers that emulates and constrains syscalls. It specifically runs on top of a container platform and k

No, I'm not. Gvisor is a security layer around Linux containers that emulates and constrains syscalls. It specifically runs on top of a container platform and kernel. What I'm suggesting is a stripped down Linux-like kernel that is really good at running exactly one process. I'm describing a microkernel.

tptacek 3y ago
gvisor emulates a big chunk of the Linux system call interface, and, remarkably, a further large chunk of Linux kernel state. It's architecturally similar to uml (though not as complete; like Firecracker, it's optimized to a specific set of workloads). gvisor is not like a seccomp-bpf process sandbox that just ACLs system calls.
- packetlost 3y ago
  Ok, I oversimplified a bit. Regardless, I'm suggesting something that still runs in emulated hardware isolation and implements drivers for Firecracker/QEMU's subset of hardware.
  - tptacek 3y ago
    gvisor does emulate some hardware. See, for instance, its network stack. At any rate: why is this better than just using KVM and Firecracker? The big problem with gvisor is that the emulation you're talking about has pretty tough overhead.
    - packetlost 3y ago
      Let's you get the best of both gvisor and Firecracker: efficient use of resources (ie. not running a full Linux kernel + scheduler, and most importantly, network stack for every lambda) while getting the isolation that comes from virtualization. You can achieve this in one of 2 ways: make a new kernel and add support for targeting it in the supported languages, or strip the Linux kernel down and reimplement the parts that aren't optimized for you short-lived VM lifecycle (scheduler, network stack, etc.).
      - tptacek 3y ago
        Stripping the Linux kernel down is what people do with Firecracker. I'm curious what savings you see in the Linux networking stack. You could compile it out and just rely on vsocks, but now you're breaking everyone's code and you're not winning anything on performance.
        packetlost 3y ago
        Perhaps I'm off base (I'm not an expert in this area), but I recall reading that one of the major challenges with Lambda was the latency that initializing the network stack introduces. Perhaps that's been solved by now, but my naive idea is to have the guest not really run it's own network stack (at least the MAC/IP portion of it) and instead delegate the entire stack (IP and all) to the virtual device, which can be implemented by Firecracker/QEMU/whatever. I guess at that point, the amount of mangling you'd need to do to the kernel probably isn't worth it and you should just use Gvisor... ah oh well. Regardless, I'm still surprised microkernels aren't more popular in this space, but perhaps the losing the ecosystem of Linux libs/applications is a non-starter. Even if the idea wasn't fruitful, the conversation was fun. Thanks for engaging and challenging my bad ideas! Edit: I've also realized I was thinking of Unikernels, not microkernels and I've been calling it the wrong thing all night. *sigh*
        monocasa 3y ago
        FWIW, Linux itself has plenty of support for TCP offload engines. I don't think Firecracker uses that at the moment, but there's no reason why it has to be that way if that's a true bottleneck in the system.
        caskstrength 3y ago
        > Linux itself has plenty of support for TCP offload engines Could you link to any specific Linux kernel source that implements support for TCP offload? AFAIK networking subsystem maintainers were always opposed to accommodate TCP offload because it is A Bad Idea.
talideon 3y ago
That's not what a microkernel is. A microkernel is a kernel that pushes services traditionally included in the kernel, such as networking, out into userspace. The closest things to what you're describing are unikernels and NetBSD rump kernels.
- packetlost 3y ago
  Yeah, further down I realized I was crossing wires and was thinking of a Unikernel. My bad!
reachableceo 3y ago
I believe the term you are looking for is unikernel, not micro kernel.
- yencabulator 3y ago
  It's a unikernel really only if you rip out any security boundaries inside the VM and link the kernel into the app. If you still maintain a syscall boundary inside the VM, it's just another kernel.

tptacek 3y ago

gvisor emulates a big chunk of the Linux system call interface, and, remarkably, a further large chunk of Linux kernel state. It's architecturally similar to uml (though not as complete; like Firecracker, it's optimized to a specific set of workloads). gvisor is not like a seccomp-bpf process sandbox that just ACLs system calls.

packetlost 3y ago
Ok, I oversimplified a bit. Regardless, I'm suggesting something that still runs in emulated hardware isolation and implements drivers for Firecracker/QEMU's subset of hardware.
- tptacek 3y ago
  gvisor does emulate some hardware. See, for instance, its network stack. At any rate: why is this better than just using KVM and Firecracker? The big problem with gvisor is that the emulation you're talking about has pretty tough overhead.
  - packetlost 3y ago
    Let's you get the best of both gvisor and Firecracker: efficient use of resources (ie. not running a full Linux kernel + scheduler, and most importantly, network stack for every lambda) while getting the isolation that comes from virtualization. You can achieve this in one of 2 ways: make a new kernel and add support for targeting it in the supported languages, or strip the Linux kernel down and reimplement the parts that aren't optimized for you short-lived VM lifecycle (scheduler, network stack, etc.).
    - tptacek 3y ago
      Stripping the Linux kernel down is what people do with Firecracker. I'm curious what savings you see in the Linux networking stack. You could compile it out and just rely on vsocks, but now you're breaking everyone's code and you're not winning anything on performance.
      - packetlost 3y ago
        Perhaps I'm off base (I'm not an expert in this area), but I recall reading that one of the major challenges with Lambda was the latency that initializing the network stack introduces. Perhaps that's been solved by now, but my naive idea is to have the guest not really run it's own network stack (at least the MAC/IP portion of it) and instead delegate the entire stack (IP and all) to the virtual device, which can be implemented by Firecracker/QEMU/whatever. I guess at that point, the amount of mangling you'd need to do to the kernel probably isn't worth it and you should just use Gvisor... ah oh well. Regardless, I'm still surprised microkernels aren't more popular in this space, but perhaps the losing the ecosystem of Linux libs/applications is a non-starter. Even if the idea wasn't fruitful, the conversation was fun. Thanks for engaging and challenging my bad ideas! Edit: I've also realized I was thinking of Unikernels, not microkernels and I've been calling it the wrong thing all night. *sigh*
        monocasa 3y ago
        FWIW, Linux itself has plenty of support for TCP offload engines. I don't think Firecracker uses that at the moment, but there's no reason why it has to be that way if that's a true bottleneck in the system.
        caskstrength 3y ago
        > Linux itself has plenty of support for TCP offload engines Could you link to any specific Linux kernel source that implements support for TCP offload? AFAIK networking subsystem maintainers were always opposed to accommodate TCP offload because it is A Bad Idea.

talideon 3y ago

That's not what a microkernel is. A microkernel is a kernel that pushes services traditionally included in the kernel, such as networking, out into userspace. The closest things to what you're describing are unikernels and NetBSD rump kernels.

packetlost 3y ago
Yeah, further down I realized I was crossing wires and was thinking of a Unikernel. My bad!

reachableceo 3y ago

I believe the term you are looking for is unikernel, not micro kernel.

yencabulator 3y ago
It's a unikernel really only if you rip out any security boundaries inside the VM and link the kernel into the app. If you still maintain a syscall boundary inside the VM, it's just another kernel.