5 ms·

I wonder why they don't expose a kernel instead of just the rootfs. It's hard to imagine a great reason. Maybe they harden their guest kernel?

JoshTriplett 5y ago
At one point, Lambda didn't expose the ability to write custom runtimes, and only provided runtimes for specific languages. People reverse-engineered those runtimes and figured out how to build custom ones. Eventually, Amazon provided a documented, well-supported way to build and run custom runtimes, but that required documenting the interfaces provided to those runtimes (e.g. environment variables and HTTP APIs instead of language-runtime-specific APIs). I'd love to see Lambda support custom kernels. That would require a similar approach: document the actual VM interface provided to the kernel, including the virtual hardware and the mechanism used to expose the Lambda API. I'd guess that they haven't yet due to a combination of 1) not enough demand for Lambda with custom kernels and 2) the freedom to modify the kernel/hardware interface arbitrarily because they control both sides of it.
- staticassertion 5y ago
  Yeah I'd bet that it's just a "haven't justified this work yet" kinda thing. We just run Firecracker VMs ourselves.
  - nuclearnice1 5y ago
    Interesting. How does that work? How do Firecracker VMs differ from containers on lambda or fargate?
nijave 5y ago
Lambda uses a stripped down Linux kernel (afaik it has some syscalls removed) The kernel surface is part of their security model. There's some details here https://www.bschaatsbergen.com/behind-the-scenes-lambda https://www.bschaatsbergen.com/behind-the-scenes-lambda E.g. exposing the kernel would undo some intentional isolation
- JoshTriplett 5y ago
  I've seen that, but I wonder to what extent they've done custom work and to what extent they've just used established Kconfig options to compile out surface area they're not using. In any case, Firecracker+Nitro seem like they'd be a sufficient security boundary.

JoshTriplett 5y ago

At one point, Lambda didn't expose the ability to write custom runtimes, and only provided runtimes for specific languages. People reverse-engineered those runtimes and figured out how to build custom ones. Eventually, Amazon provided a documented, well-supported way to build and run custom runtimes, but that required documenting the interfaces provided to those runtimes (e.g. environment variables and HTTP APIs instead of language-runtime-specific APIs). I'd love to see Lambda support custom kernels. That would require a similar approach: document the actual VM interface provided to the kernel, including the virtual hardware and the mechanism used to expose the Lambda API. I'd guess that they haven't yet due to a combination of 1) not enough demand for Lambda with custom kernels and 2) the freedom to modify the kernel/hardware interface arbitrarily because they control both sides of it.

staticassertion 5y ago
Yeah I'd bet that it's just a "haven't justified this work yet" kinda thing. We just run Firecracker VMs ourselves.
- nuclearnice1 5y ago
  Interesting. How does that work? How do Firecracker VMs differ from containers on lambda or fargate?

nijave 5y ago

Lambda uses a stripped down Linux kernel (afaik it has some syscalls removed) The kernel surface is part of their security model. There's some details here https://www.bschaatsbergen.com/behind-the-scenes-lambda https://www.bschaatsbergen.com/behind-the-scenes-lambda E.g. exposing the kernel would undo some intentional isolation

JoshTriplett 5y ago
I've seen that, but I wonder to what extent they've done custom work and to what extent they've just used established Kconfig options to compile out surface area they're not using. In any case, Firecracker+Nitro seem like they'd be a sufficient security boundary.