7 ms·
Firecracker MicroVMs
- staticassertion 5y agoFor those using firecracker, what's your method for building a rootfs?
- progamer2018 5y agosee document here https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md https://github.com/firecracker-microvm/firecracker/blob/main...
- deivid 5y agoI build them from docker images: https://blog.davidv.dev/docker-based-images-on-baremetal.html https://blog.davidv.dev/docker-based-images-on-baremetal.htm...
- deleted 5y ago[deleted]
- lgierth 5y agoI build them using Buildah, then mkfs.ext4, mount, cp, umount. It's a little bit annoying that I'm still using root at least for the mount part. OpenWrt's build system has a method of building rootfs ext4 and squashfs images without any root, it's somewhere in that large Makefile mess. (sorry, reposting this as I first replied to the wrong parent)
- mikepurvis 5y agoI similarly have built bootable disk images with various tools including buildah and have never been able to fully get away from needing root for various chrooty/loopbacky parts of the process. In principle, it should be very possible to point grub at a filesystem-in-a-file and be like "install to that", but I could never make it happen; it always wanted to be trying to infer things about how to configure itself from examining the host system. And yes, I've studied the OpenWRT build to no avail. I would be delighted for someone to dissect whatever it is that goes on in there and write it up.
- bminor13 5y agoNo experience with Firecracker specifically, but if squashfs images are sufficient, one should be able to build a tar archive of the filesystem without root (where all the files have the correct owners, mode, etc.) and then convert it to squashfs using `tar2sqfs` in https://github.com/AgentD/squashfs-tools-ng https://github.com/AgentD/squashfs-tools-ng, also without root - I've done something like this to create squashfs images in constrained build environments, which worked well.
- rad_gruchalski 5y agoDirectly from Docker image: https://github.com/combust-labs/firebuild#high-level-example https://github.com/combust-labs/firebuild#high-level-example. I’m the author.
- gavinray 5y agoThis is incredibly interesting. Is it alright if I email you with some questions?
- rad_gruchalski 5y agoSure. My email address: radek@gruchalski.com.
- ignoramous 5y agonb, building from docker is also what fly.io does: https://news.ycombinator.com/item?id=22514017 https://news.ycombinator.com/item?id=22514017
- rad_gruchalski 5y agoYes, I can also build from Dockerfile: sudo $GOPATH/bin/firebuild rootfs \ --profile=standard \ --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \ --cni-network-name=machine-builds \ --ssh-user=alpine \ --vmlinux-id=vmlinux-v5.8 \ --tag=combust-labs/consul:1.9.4 That SSH bit is no longer required as I'm using MMDS instead.
- tptacek 5y agoWe're fancier now, and use containerd with LVM. https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
- ignoramous 5y agoThat's super fancy, indeed. Thanks for these write-ups. They make for a riveting read, even when more than half of it usually is beyond my technical know-how.
- dang 5y agoA past thread: Firecracker: Secure and fast microVMs for serverless computing - https://news.ycombinator.com/item?id=22512196 https://news.ycombinator.com/item?id=22512196 - March 2020 (103 comments)
- sudhirj 5y agoIs there something new that prompted this posting?
- capableweb 5y agoIf I were to guess, I'd guess because Firecracker is something "that good hackers would find interesting" https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- jrk 5y agoI think the question is raised because Firecracker was released years ago and has been posted and discussed many times before.
- tlrobinson 5y ago“Are reposts ok?” “If a story has not had significant attention in the last year or so, a small number of reposts is ok. Otherwise we bury reposts as duplicates.” https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
- weikju 5y agoI think the question is more along the lines of - was there another news, recently posted, that prompted this post? Like for example if fly.io was announced yesterday, and they said they used Firecracker, then someone would submit this link about Firecracker. So the original poster in this thread was wondering if there was something to that effect. Not decrying something being posted that was not new or already discussed before...
- moondev 5y agoWould be interesting to see a project that unveils these "chain of influences" for tracking HN engagement on stories or even comments
- m00dy 5y agoanyone knows this can create gpu-enabled microvms ?
- tptacek 5y agoThis is tricky for Firecracker, both because of security (and the attendant switching costs of keeping exactly one VM connected to a GPU at a time) and because of the Firecracker tenet of supporting oversubscription. They've been talking about it for a couple years. I think they're actually talking about forking off a GPU-capable variant of the project to work around the drama.
- lgierth 5y agoNo support for PCI devices at the moment, but I believe it's planned for some point in the future.
- wmf 5y agoMaybe try Cloud Hypervisor instead. https://www.phoronix.com/scan.php?page=news_item&px=Cloud-Hypervisor-0.13 https://www.phoronix.com/scan.php?page=news_item&px=Cloud-Hy...
- sudhirj 5y agoBest example of current usage that comes to mind (outside of AWS’s own Lambda & Fargate) is Fly.io - they they use Firecracker for all the containers.
- ctvo 5y agoAdditionally, I remember browsing through BuildBuddy's project and saw they used it for build run isolation in their enterprise offering. - https://www.buildbuddy.io https://www.buildbuddy.io - https://github.com/buildbuddy-io/buildbuddy/blob/master/enterprise/server/remote_execution/containers/firecracker/firecracker.go https://github.com/buildbuddy-io/buildbuddy/blob/master/ente...
- rad_gruchalski 5y agoThere’s also Koyeb. I’m not affiliated.
- slimsag 5y agoWe also use it at Sourcegraph (to run arbitrary LSIF code indexers in a sandboxed environment)
- tapirl 5y agoFrom their docs, their product looks amazing. However I haven't successfully deployed my app on their platform: https://community.fly.io/t/first-app-fails-to-deploy/2754/3 https://community.fly.io/t/first-app-fails-to-deploy/2754/3 This might be related to my network problem, I will try other deploy ways when I get some spare time.
- paxys 5y agoDoes anyone know of any good orchestrators for Firecracker VMs? While the technology is great, the CLI and API are still a bit too complex and low-level to implement some basic use cases in production.
- rad_gruchalski 5y agoI did this some months ago: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild. Plan is to come back to it sometime towards the end of the year.
- badLiveware 5y agoKubernetes using Kata containers[1] as a containerd[2] backend [1] https://aws.amazon.com/blogs/opensource/kata-containers-1-5-firecracker-support/ https://aws.amazon.com/blogs/opensource/kata-containers-1-5-... [2] https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md https://github.com/kata-containers/kata-containers/blob/main...
- adolph 5y agoHow does that compare to firecracker-containerd? https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container... This repository enables the use of a container runtime, containerd, to manage Firecracker microVMs. Like traditional containers, Firecracker microVMs offer fast start-up and shut-down and minimal overhead. Unlike traditional containers, however, they can provide an additional layer of isolation via the KVM hypervisor.
- NathanKP 5y agoTake a look at Weaveworks Ignite for an open source tool for running Firecracker VMs from container images: https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite And of course AWS Fargate and Lambda use this tech under the hood transparently, so that's always an option if you don't want to host and operate it yourself.
- unixhero 5y agoAw geez. Seems like we're going to need a Firecracker awesomelist now then!
- imachine1980_ 5y agowhat is the advantage of server-less if you have to have to manage the servers?
- jjice 5y agoThis is more meant for providers. Best example being AWS's serverless offerings, as well as fly.io.
- staticassertion 5y agoIf you're hosting someone else's code.
- hinkley 5y agoIf you work at a large enough company it all starts to look like someone else’s code. The malicious person can always find the worst way to break everything, but dumb luck can get you pretty close.
- gizdan 5y agoAnyone any thought on using this in a homelab (just random servers such as laptops and microservers) with K8s vs normal containers?
- tough 5y agoI bookmarked this about firecracker's internals in HN in the past https://www.talhoffman.com/2021/07/18/firecracker-internals/ https://www.talhoffman.com/2021/07/18/firecracker-internals/
- andrewstuart 5y agoNote that this Amazon developed technology does not work on Amazon Web Services instances. It does however work on Google, Azure and Digital Ocean instances where you can configure nested virtualisation.
- cptnntsoobv 5y agoIt works on AWS's metal instances, doesn't it? That is definitely a constraint though, as the smallest available metal instances are pretty large, hence costly on a per instance basis.
- oneearedrabbit 5y agoTechnically speaking you are right. If you are not limited to AWS, then even a $5 DigitalOcean instance is be able to host 4-5 VMs.
- paxys 5y agoIt does work on AWS on the instances that support nested virtualization, same as every other provider.
- cutemonster 5y ago> does not work on Amazon Web Services instances Why not? And what is Google, Azure and DI doing differently
- jsolson 5y agoFirecracker relies on KVM in the Linux kernel which relies on VT-x or similar. Offering VT-x inside a VM that's already using VT-x is called nested virtualization. EC2 VMs do not support nested virtualization, while others do. That said, it works fine on EC2 Bare Metal instances. (n.b., I work on the virtualization stack on Google Compute Engine)
- bushbaba 5y agoWouldn’t you want to run firecracker on bare metal anyways. Nested VMs come with a decent perf hit.
- mastax 5y agoI just deployed a WordPress container on the same host that runs everything else. Makes me a bit uncomfortable. Was thinking about running a VM just as a Docker Swarm worker node, but this post reminded me that these virtualization runtimes exist. Looks like there's a drop-in containerd replacement using firecracker. Kata containers look a bit better supported, though? Or am I too paranoid about container escapes?
- throwaway984393 5y agoWhat are you worried about them doing? If they take over the app inside the container, they can access your database and seed malware in your users' browsers.
- staticassertion 5y agoI don't think you can be too paranoid about container escapes. We actually don't consider containers to be a reliable security boundary - we take it where we can, but if we need real isolation it has to be combined with another approach.