18 ms·

Ignite – Use Firecracker VMs with Docker images

234 points by cercatrova 4y ago

cercatrova 4y ago
Related, My VM is Lighter (and Safer) than your Container (2017): https://news.ycombinator.com/item?id=32764501 https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.
- deathanatos 4y ago
  What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.
  - russell_h 4y ago
    Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
    - easton 4y ago
      15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
      - jakogut 4y ago
        There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
    - deathanatos 4y ago
      Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)
  - sofixa 4y ago
    Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).
  - rwmj 4y ago
    You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.
  - jeroenhd 4y ago
    The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.
- topspin 4y ago
  I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't even live migrate a firecracker VM right now. So yes, VMs are great and will ultimately resurge to the detriment of 'everything is a container' serverless orchestration, but Firecracker in its current form won't cut it. A gulf exists between giant cloud operators and their lucrative 'serverless' (only) model and everyone else. Everyone else must avoid the insurmountable costs associated with abandoning the immense value that stateful VMs provide and the minute something emerges that delivers a seamless VM orchestration system it will skyrocket.
  - tptacek 4y ago
    Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
    - 0xbadcafebee 4y ago
      From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".
noveltyaccount 4y ago
> Ignite makes Firecracker easy to use by adopting its developer experience from containers. With Ignite, you pick an OCI-compliant image (Docker image) that you want to run as a VM, and then just execute ignite run instead of docker run. Damn, that's quite a sales pitch. Excited to follow along!
mbreese 4y ago
For me, this is the deal breaker, but this is still exciting to me: > Note: At the moment ignite and ignited need root privileges on the host to operate due to certain operations (e.g. mount). This will change in the future. I’d love it if this ends up getting changed. It’s a hard line to walk, but this would make Docker containers feasible on multi-user systems. (Without using Singularity, which has its own downsides).
- Atreiden 4y ago
  Found the HPC guy. I was so excited for this until I got to that line. I'll still be following this closely though, the potential is huge.
  - mbreese 4y ago
    So, you’ve been waiting for something like this too? I’d love to be able to submit Slurm jobs backed by a Docker container. That would be a reproducibility dream for me. And make it so I didn’t have to install so much software or still maintain Environment modules. As it is, I can get partially there with Singularity, which isn’t terrible.
    - Atreiden 4y ago
      Absolutely. With firecracker, AWS seems to have solved the multi-tenancy problem that has plagued HPC since its inception. I've just been waiting for some way to implement that in a sane way for on-prem infrastructure. If not for the root requirement, this would be perfect. Singularity does get us part of the way there, but it's just another abstraction layer that researchers have to work around. And it's far from perfect.
solarkraft 4y ago
This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 https://github.com/weaveworks/ignite/issues/874. You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers are definitely a great idea; another project aiming to do it (a little differently) is Kata Containers. It has a lot more industry support, can also run on Firecracker (though QEMU microVMs are just as good) and can (theoretically, I haven't gotten it to work) interact with many container runtimes that are not Docker (the focus appears to be on Kubernetes). While more powerful, Kata Containers are also much, much more complicated (and honestly under- and misleadingly documented). Ignite could fill the role of something simpler and more easy to get started with. But please, if you aim to attract Docker users, actually make it compatible with Docker principles. There's still a lot of unused potential in containers as VMs; Kata+Kubernetes probably currently realizes the most of it.
- claytongulick 4y ago
  > microVMs for running containers are definitely a great idea I think my greybeard must be getting too long... This sounds like insanity to me. Why not just make a base image and some simple setup scripts? If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds. [1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-seconds https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...
  - sofixa 4y ago
    > Why not just make a base image and some simple setup scripts? That makes it possible for you to create an image (VM or container?, good start. But what Firecracker and containers solve is isolation and ease of distribution.
    - pjmlp 4y ago
      Just like VM repos used to be used....
Spivak 4y ago
Damn, this is going to eat the world. Docker but with full host isolation making it suitable for untrusted workloads, hot damn. This could for real make fully managed multi-tenant completely serverless k8s possible once it gets to the point where it can be used as a containerd and CRI implementation.
- NexRebular 4y ago
  Been doing that by running kubernetes inside SmartOS zones via bhyve VMs on multi-tenant systems. Works pretty well, although they are slightly more resource hungry VMs than the ones in this article.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
vajrabum 4y ago
The major reason for the initial excitement about containers was the relative efficiency as compared to VMs. I understand that Firecracker has a low memory overhead and a fast startup time but I'd be interested to know what the virtualization overhead is in terms of cpu performance. I poked around on google a bit for a few minutes but didn't find anything.
- phamilton 4y ago
  My understanding is that it's just KVM under the hood, so cpu perf should match KVM. Firecracker is mostly just optimizing startup time and memory overhead.
duskwuff 4y ago
I like the way this sounds, but... is the project still alive? There haven't been any commits to the main branch since July, and the last release was over a year ago.
- tillk 4y ago
  I have the same question. WeaveWorks has another project called footloose which allows docker containers to be used like VMs. Also dead. The only other project/company (aside from Kata) promoting firecracker was Talos. But not sure if they maintain their integration still. https://www.talos.dev/v1.2/reference/cli/#talosctl-cluster-create https://www.talos.dev/v1.2/reference/cli/#talosctl-cluster-c...
  - expertsteve 4y ago
    Talos no longer supports creating a cluster automatically in a firecracker VM (which you can still do in docker, via `talosctl cluster create`) but you can certainly run Talos Linux in firecracker VMs and create a cluster that way.
    - tillk 4y ago
      Interesting. So Talos is both a Linux and a k8s distribution? I am less interested in running k8s itself in Firecracker (or kata, or similar), but the pods started by it. I think there are advantages to that.
zozbot234 4y ago
Running Docker in a VM is not new at all. This is even how Docker for Windows/Mac works. The GitHub page states that Ignite does more than just "wrapping a container in a VM layer" but I'm not sure how that matters, since the notion of a "container" is purely about configuration. It seems like a distinction without a difference.
- matharmin 4y ago
  Docker for Windows/Mac uses a single VM for all your containers. Firecracker/Ignite has a dedicated micro-VM for each container - a completely different architecture, with much better isolation between containers. And the 125ms boot time is not something you get with a typical VM. But I do agree that they don't explain how Ignite is different from "wrapping a container in a VM layer" that Kata/gVisor does.
- ritchiey 4y ago
  Ignite is running a VM per container. Docker for Windows/Mac runs all containers in a single VM.
nailer 4y ago
Can someone help me understand whether this is: - My app per the Dockerfile - Container - Firecracker VM Or: - My app per the Dockerfile - Firecracker VM Thanks!
- NathanKP 4y ago
  The latter. If you typed `docker ps` on your host there would be nothing showing up. The application is running inside of a micro VM. It should also be noted that containerized processes that are running on a host will still show up on the host's process tree if you look at the output of `ps`, `htop` or similar commands. Containers aren't real boundaries, just namespaces on the host machine. This is the not the case when using Ignite, as the process from the container image is running in an actual micro VM, rather than running on the host inside of a namespace
  - nailer 4y ago
    Thanks for the answer. Oh then this is awesome. The convenience of Docker images with the proper isolation of MicroVMs.
icybox 4y ago
Does this work for anybody? I've just tried it in a KVM Debian 11 VM on proxmox. Deploying from yml manifest does not like my `ssh: true', removing existing /etc/firecracker/manifest/*.yml file segfaults the daemon. Trying to `ignite run ubuntu:latest` or `ignite run alpine:latest` dies on trying to run init ... where's the benefit of leveraging existing containers? Looks abandoned for a year at least ...
Kalanos 4y ago
can someone help me understand why VMs are necessary here?
- lbotos 4y ago
  "Isolation" right? By making that boundary, you get "slightly safer" code execution in a multi-tenant environment.
  - Kalanos 4y ago
    ah okay. kind of a brute force way to go about it. why bother with containers at that point? just use VM images

234 points by cercatrova 4y ago

Related, My VM is Lighter (and Safer) than your Container (2017): https://news.ycombinator.com/item?id=32764501 https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.

deathanatos 4y ago
What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.
- russell_h 4y ago
  Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
  - easton 4y ago
    15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
    - jakogut 4y ago
      There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
  - deathanatos 4y ago
    Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)
- sofixa 4y ago
  Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).
- rwmj 4y ago
  You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.
- jeroenhd 4y ago
  The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.
topspin 4y ago
I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't even live migrate a firecracker VM right now. So yes, VMs are great and will ultimately resurge to the detriment of 'everything is a container' serverless orchestration, but Firecracker in its current form won't cut it. A gulf exists between giant cloud operators and their lucrative 'serverless' (only) model and everyone else. Everyone else must avoid the insurmountable costs associated with abandoning the immense value that stateful VMs provide and the minute something emerges that delivers a seamless VM orchestration system it will skyrocket.
- tptacek 4y ago
  Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
  - 0xbadcafebee 4y ago
    From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".

noveltyaccount 4y ago

> Ignite makes Firecracker easy to use by adopting its developer experience from containers. With Ignite, you pick an OCI-compliant image (Docker image) that you want to run as a VM, and then just execute ignite run instead of docker run. Damn, that's quite a sales pitch. Excited to follow along!

mbreese 4y ago

For me, this is the deal breaker, but this is still exciting to me: > Note: At the moment ignite and ignited need root privileges on the host to operate due to certain operations (e.g. mount). This will change in the future. I’d love it if this ends up getting changed. It’s a hard line to walk, but this would make Docker containers feasible on multi-user systems. (Without using Singularity, which has its own downsides).

Atreiden 4y ago
Found the HPC guy. I was so excited for this until I got to that line. I'll still be following this closely though, the potential is huge.
- mbreese 4y ago
  So, you’ve been waiting for something like this too? I’d love to be able to submit Slurm jobs backed by a Docker container. That would be a reproducibility dream for me. And make it so I didn’t have to install so much software or still maintain Environment modules. As it is, I can get partially there with Singularity, which isn’t terrible.
  - Atreiden 4y ago
    Absolutely. With firecracker, AWS seems to have solved the multi-tenancy problem that has plagued HPC since its inception. I've just been waiting for some way to implement that in a sane way for on-prem infrastructure. If not for the root requirement, this would be perfect. Singularity does get us part of the way there, but it's just another abstraction layer that researchers have to work around. And it's far from perfect.

solarkraft 4y ago

This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 https://github.com/weaveworks/ignite/issues/874. You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers are definitely a great idea; another project aiming to do it (a little differently) is Kata Containers. It has a lot more industry support, can also run on Firecracker (though QEMU microVMs are just as good) and can (theoretically, I haven't gotten it to work) interact with many container runtimes that are not Docker (the focus appears to be on Kubernetes). While more powerful, Kata Containers are also much, much more complicated (and honestly under- and misleadingly documented). Ignite could fill the role of something simpler and more easy to get started with. But please, if you aim to attract Docker users, actually make it compatible with Docker principles. There's still a lot of unused potential in containers as VMs; Kata+Kubernetes probably currently realizes the most of it.

claytongulick 4y ago
> microVMs for running containers are definitely a great idea I think my greybeard must be getting too long... This sounds like insanity to me. Why not just make a base image and some simple setup scripts? If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds. [1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-seconds https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...
- sofixa 4y ago
  > Why not just make a base image and some simple setup scripts? That makes it possible for you to create an image (VM or container?, good start. But what Firecracker and containers solve is isolation and ease of distribution.
  - pjmlp 4y ago
    Just like VM repos used to be used....

Spivak 4y ago

Damn, this is going to eat the world. Docker but with full host isolation making it suitable for untrusted workloads, hot damn. This could for real make fully managed multi-tenant completely serverless k8s possible once it gets to the point where it can be used as a containerd and CRI implementation.

NexRebular 4y ago
Been doing that by running kubernetes inside SmartOS zones via bhyve VMs on multi-tenant systems. Works pretty well, although they are slightly more resource hungry VMs than the ones in this article.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

vajrabum 4y ago

The major reason for the initial excitement about containers was the relative efficiency as compared to VMs. I understand that Firecracker has a low memory overhead and a fast startup time but I'd be interested to know what the virtualization overhead is in terms of cpu performance. I poked around on google a bit for a few minutes but didn't find anything.

phamilton 4y ago
My understanding is that it's just KVM under the hood, so cpu perf should match KVM. Firecracker is mostly just optimizing startup time and memory overhead.

duskwuff 4y ago

I like the way this sounds, but... is the project still alive? There haven't been any commits to the main branch since July, and the last release was over a year ago.

tillk 4y ago
I have the same question. WeaveWorks has another project called footloose which allows docker containers to be used like VMs. Also dead. The only other project/company (aside from Kata) promoting firecracker was Talos. But not sure if they maintain their integration still. https://www.talos.dev/v1.2/reference/cli/#talosctl-cluster-create https://www.talos.dev/v1.2/reference/cli/#talosctl-cluster-c...
- expertsteve 4y ago
  Talos no longer supports creating a cluster automatically in a firecracker VM (which you can still do in docker, via `talosctl cluster create`) but you can certainly run Talos Linux in firecracker VMs and create a cluster that way.
  - tillk 4y ago
    Interesting. So Talos is both a Linux and a k8s distribution? I am less interested in running k8s itself in Firecracker (or kata, or similar), but the pods started by it. I think there are advantages to that.

zozbot234 4y ago

Running Docker in a VM is not new at all. This is even how Docker for Windows/Mac works. The GitHub page states that Ignite does more than just "wrapping a container in a VM layer" but I'm not sure how that matters, since the notion of a "container" is purely about configuration. It seems like a distinction without a difference.

matharmin 4y ago
Docker for Windows/Mac uses a single VM for all your containers. Firecracker/Ignite has a dedicated micro-VM for each container - a completely different architecture, with much better isolation between containers. And the 125ms boot time is not something you get with a typical VM. But I do agree that they don't explain how Ignite is different from "wrapping a container in a VM layer" that Kata/gVisor does.
ritchiey 4y ago
Ignite is running a VM per container. Docker for Windows/Mac runs all containers in a single VM.

nailer 4y ago

Can someone help me understand whether this is: - My app per the Dockerfile - Container - Firecracker VM Or: - My app per the Dockerfile - Firecracker VM Thanks!

NathanKP 4y ago
The latter. If you typed `docker ps` on your host there would be nothing showing up. The application is running inside of a micro VM. It should also be noted that containerized processes that are running on a host will still show up on the host's process tree if you look at the output of `ps`, `htop` or similar commands. Containers aren't real boundaries, just namespaces on the host machine. This is the not the case when using Ignite, as the process from the container image is running in an actual micro VM, rather than running on the host inside of a namespace
- nailer 4y ago
  Thanks for the answer. Oh then this is awesome. The convenience of Docker images with the proper isolation of MicroVMs.

icybox 4y ago

Does this work for anybody? I've just tried it in a KVM Debian 11 VM on proxmox. Deploying from yml manifest does not like my `ssh: true', removing existing /etc/firecracker/manifest/*.yml file segfaults the daemon. Trying to `ignite run ubuntu:latest` or `ignite run alpine:latest` dies on trying to run init ... where's the benefit of leveraging existing containers? Looks abandoned for a year at least ...

Kalanos 4y ago

can someone help me understand why VMs are necessary here?

lbotos 4y ago
"Isolation" right? By making that boundary, you get "slightly safer" code execution in a multi-tenant environment.
- Kalanos 4y ago
  ah okay. kind of a brute force way to go about it. why bother with containers at that point? just use VM images