13 ms·
Firecracker internals: Inside the technology powering AWS Lambda (2021)
- mjb 4y agoGreat overview! > It is highly recommended to read the source code of this amazing project and explore it yourselves - https://github1s.com/firecracker-microvm/firecracker https://github1s.com/firecracker-microvm/firecracker. Yes! The Firecracker source is super readable, and a great way to learn about this stuff in detail. There's very little magic there, partially thanks to the efforts of the team to keep things accessible and well documented, and partially thanks to how Linux's KVM APIs abstract away some of the hard and hardware-dependent stuff. For more on the context of how we use Firecracker at AWS, check out: - Our NSDI'20 paper, on Firecracker and Lambda: https://www.usenix.org/conference/nsdi20/presentation/agache https://www.usenix.org/conference/nsdi20/presentation/agache - Chris and Julien talking about Lambda internals at reInvent'22 (including details of the Firecracker-derived technology behind "snapstart"): https://www.youtube.com/watch?v=EplOzQqgstA https://www.youtube.com/watch?v=EplOzQqgstA - "Lightweight Virtualization, Opportunities and Challenges", a talk I gave in 2020 which covers some serverless/Firecracker topics: https://www.youtube.com/watch?v=ADOfX2LiEns https://www.youtube.com/watch?v=ADOfX2LiEns - The Security Overview of AWS Lambda, which looks at this stuff from the security perspective: https://docs.aws.amazon.com/whitepapers/latest/security-overview-aws-lambda/security-overview-aws-lambda.html https://docs.aws.amazon.com/whitepapers/latest/security-over...
- nighthawk454 4y agoThanks for linking these additional talks! Tangent but that github1s link from the OP (to an in-browser VS Code instance) makes a clear call out that it's not officially from GitHub. Which is fine. But for completion, GitHub does have an official feature for this by changing the url to `github.dev` or pressing '.' on any GitHub repo: https://github.dev/firecracker-microvm/firecracker https://github.dev/firecracker-microvm/firecracker
- mdaniel 4y agoWhile github.dev may be better in some circumstances, github1s works in an incognito window, without being logged in to GitHub; the github.dev version appears briefly to boot up, then redirects to the login page for GitHub
- rurban 4y agoIs this from Dresden or from Seattle or some other kernel team?
- oblio 4y agoBucharest/Iași, Romania. https://ro.linkedin.com/in/raduweiss https://ro.linkedin.com/in/raduweiss They have quite massive R&D offices in Romania, several thousand folks there.
- rurban 4y agoWhow, impressive! I visited the Iasi math university years ago. They were very good then. Eg lot of Prolog, where others still did Java or Pascal.
- Cthulhu_ 4y agoDoes making it open source mean you get a lot of competition? I believe Cloudflare's lambda offerings are pretty popular for example. Then again, large companies that already invested in AWS are probably not likely to switch providers for specific workloads like lambdas.
- tdullien 4y agoIirc it's based on crosvm, so AWS didn't do much "make it open source" as "adapted an existing piece of open source".
- eloff 4y agoCloudflare uses V8 for sandboxing, not virtual machines. It’s much lighter weight, but it’s also limited to just what V8 can run (JavaScript and web assembly). Quite a different offering to lambda. It shares no source in common.
- nimbius 4y agocurious: for the blockIO engine can a qcow2 image be specified as a path? how do i run python in this? can i get to the BPF engine?
- di456 4y agoFly.io is also running firecracker vm's
- tptacek 4y agoWe do very little interesting stuff on the VMM side; this is an Amazon project we're happy to draft off of.
- nailer 4y agoThat’s cool. As an old school unix sysadmin that’s aware of the “containers don’t contain”, people ditching proper isolation for containers everywhere for performance reasons has been alarming. now with Firecracker we have isolation and performance.
- donmcronald 4y agoI’ve always thought Firecracker would be great for CI runners.
- maccard 4y agoI agree. I prototyped integrating AWS Lambda into our CI pipeline last year, and was very impressed. Unfortunately, I hit a wall when it came to limiting concurrent executing lambdas (which isn't firecracker's fault),but it was very promising up to then. I'm also curious about using fly.io's machines API [0] for the same thing. [0] https://fly.io/docs/machines/ https://fly.io/docs/machines/
- scarface74 4y agoThis is basically what CodeBuild does. The default Docker containers that CodeBuild uses (you can create your own) and the shell script it uses to parse the yaml configuration file (mostly a list of shell scripts) are all open source and the entire process can be run locally. https://github.com/aws/aws-codebuild-docker-images https://github.com/aws/aws-codebuild-docker-images https://docs.aws.amazon.com/codebuild/latest/userguide/use-codebuild-agent.html https://docs.aws.amazon.com/codebuild/latest/userguide/use-c... Disclaimer: I work for AWS. But nowhere near the team that developed Firecracker
- throwawayacc5 4y agoAny tooling like Docker/Podman to make it as easy to kick off Firecracker VMs as it is to kick off containers?
- kohlerm 4y agohttps://github.com/weaveworks/ignite https://github.com/weaveworks/ignite ?
- wanderingmind 4y agoDoesn't seem to be actively developed. Last commit is over 4 months
- Strum355 4y agoIt's pretty much unmaintained, but it does still work!
- kohlerm 4y agoYeah, I think it's strange that no one wants to improve on something like this. I can see a lot of use cases where you would want to just use one kubernetes cluster but have support for multi-tenancy with pods being isolated.
- klavinski 4y agoNot a drop-in replacement: the OCI image entry point is not automatically executed. Issue opened in 2021: https://github.com/weaveworks/ignite/issues/874 https://github.com/weaveworks/ignite/issues/874
- deleted 4y ago[deleted]
- zokier 4y agoKata containers https://katacontainers.io/ https://katacontainers.io/
- weejewel 4y agoSuper interesting read! I’ve been playing with Firecracker on a Raspberry Pi 4, but never could get Docker inside a Firecracker uVM to work. Should this be supported at all?
- vbitz 4y agoDepends how your starting the VM. I’ve run Docker on Firecracker with a Raspberry PI 4 before but it needed some fixes. One possibly is if your running directly from a Initramfs without a block device then docker needs DOCKER_RAMDISK set as a environment variable. Otherwise it’s possible the minimal kernel your Firecracker config uses doesn’t support it out of the box. You can use a regular kernel but you need to make sure modules can be loaded from somewhere.
- weejewel 4y agoThanks for your reply! Do you have a guide or more information somewhere? I found that while Firecracker seems popular, it's sparsely documented.
- talhof8 4y agoOG here. Thanks for posting! Can’t believe it got to 5th place on HN :)
- qhwudbebd 4y agoGreat write-up! One extra detail that might be worth including in the block device/storage section is that Firecracker doesn't yet pass through discard (trim) from guest to host (as fallocate(FALLOC_FL_PUNCH_HOLE) if file-backed) like qemu? The fast/efficient asynchronous io_uring backend now in released Firecracker makes it a good alternative for full-sized VMs as well as the original short-lived micro-VM use.
- natrys 4y agoI discovered the other day that the LXC project CI produces rootfs for many interesting distros [1]. Makes it very convenient to spin up a firecracker VM. [1] https://uk.lxd.images.canonical.com/images/ https://uk.lxd.images.canonical.com/images/
- dennisy 4y agoGreat read! Thank you! Does anyone know what open source competitors exist? For example is there a GCP version to check out and compare? There are many companies offering such services, would be interesting to know who is using what!
- jsty 4y agoAn analogous project from Google with similar use cases is gvisor, which IIRC underlies Cloud Run: https://gvisor.dev/ https://gvisor.dev/
- anon223345 4y agoLegit wondering why under the hood, how come Google Cloud Functions handle promises and promise chaining so much better than AWS Lambda I have to offload requests to state machines in AWS Lambdas which is a pain when prototyping
- anonymousDan 4y agoCan you expand on what you mean by this? You mean that Lambda forces you into an async style of programming where you queue events to handle completion of long running subrequests?
- deleted 4y ago[deleted]
- anon223345 4y agoActually what I’m saying is lambdas, when doing tens of thousands of invocations, will handle async requests much worse than GCP Cloud Functions or KNative Basically the lambdas, even when you return promises, or set long timeouts, will conclude before completing a request If you’re doing 5 lambda invocations you don’t really see this, but when you’re doing a lot of them you can see in the statistics a huge failure compared to GCloud Functions UNLESS you offload each fetch to its own state machines, which is technically the correct way to do it but is way over engineered compared to GCloud
- touisteur 4y agoQuestions for the firecracker users out there: what version of the kernel do you use (the github page says 5.10 but isn't that quite old?) what (extremely minimal, I imagine?) kernel configuration What do you use to build the 'micro' images (I'm guessing many won't even have a classic pid-1 such as systemd, but put their software as pid-1?). How do you keep timesync of you're not using a timesync daemon? Can you make one of these daemons work on af_vsock (I know firecracker gives a virtio-backed Ethernet device but what if you only want af_vsocks?). Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
- CompuIves 4y agoAt CodeSandbox we use Firecracker to run our VMs (more info here: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-seconds https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...). To answer the questions: > what version of the kernel do you use (the github page says 5.10 but isn't that quite old?) Right, they have tested with 5.10, but it also works with higher kernel versions. Our host currently runs 5.19 and we're planning to upgrade to 6.1 soon. The guest runs 5.15.63, we use a config very similar to the recommended config by FC team (it's in the FC repo). It's important to mention that we had to disable async pagefaulting (a KVM feature) with more modern kernel versions, as VMs could get stuck waiting for an PF resolve. > What do you use to build the 'micro' images We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk. For the init, we forked the open sourced init from the Fly team (https://github.com/superfly/init-snapshot https://github.com/superfly/init-snapshot) and changed/added some functionality. > How do you keep timesync of you're not using a timesync daemon? IIRC we expose the time as a PTP device (handled by kvm) and run phc2sys to sync the time in an interval. Firecracker has some documentation on this, where it recommends chrony. It can also be done with vsock, but it would be more manual. > Handle kernel and app logs without adding an log daemon, and same through vsocks, etc? The init forwards stdout/stderr of the command it runs to its own stdout, which Firecracker then logs out by itself. A supervisor reads these and writes the logs to files.
- easton 4y ago
- mlejva 4y agoIf you think Firecracker is interesting, checkout Cloud Hypervisor [0]. The difference according to Cloud Hypervisor team: > A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm project's implementations. Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor. > The goal of the Cloud Hypervisor project differs from the aforementioned projects in that it aims to be a general purpose VMM for Cloud Workloads and not limited to container/serverless or client workloads. Firecracker is such a great piece of technology. I'm amazed that AWS actually open-sourced it. All kudos to them. We're using Firecracker at our company to allow API companies build interactive demos like this one we built for Prisma [1]. [0] https://github.com/cloud-hypervisor/cloud-hypervisor https://github.com/cloud-hypervisor/cloud-hypervisor [1] https://playground.prisma.io https://playground.prisma.io
- tptacek 4y agoThey're both built on the same rust-vmm crates, the code is very similar, as are the interfaces. Both very cool projects.
- belter 4y agoI like this one about some of the design decisions and development challenges. "[2019] Firecracker: Lessons from the Trenches by Andreea Florescu and Alexandra Iordache" - https://youtu.be/yULy6IFy49o https://youtu.be/yULy6IFy49o Image is out of focus for most of the presentation, but not the slides and audio is good.
- ramesh31 4y agoDoes anyone know of a similar explanation for S3?