5 ms·

Related, My VM is Lighter (and Safer) than your Container (2017): https://news.ycombinator.com/item?id=32764501 https://news.ycombinator.com/item?id=32764501 L

Related, My VM is Lighter (and Safer) than your Container (2017): https://news.ycombinator.com/item?id=32764501 https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.

deathanatos 4y ago
What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.
- russell_h 4y ago
  Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
  - easton 4y ago
    15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
    - jakogut 4y ago
      There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
  - deathanatos 4y ago
    Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)
- sofixa 4y ago
  Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).
- rwmj 4y ago
  You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.
- jeroenhd 4y ago
  The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.
topspin 4y ago
I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't even live migrate a firecracker VM right now. So yes, VMs are great and will ultimately resurge to the detriment of 'everything is a container' serverless orchestration, but Firecracker in its current form won't cut it. A gulf exists between giant cloud operators and their lucrative 'serverless' (only) model and everyone else. Everyone else must avoid the insurmountable costs associated with abandoning the immense value that stateful VMs provide and the minute something emerges that delivers a seamless VM orchestration system it will skyrocket.
- tptacek 4y ago
  Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
  - 0xbadcafebee 4y ago
    From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".

deathanatos 4y ago

What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.

russell_h 4y ago
Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
- easton 4y ago
  15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
  - jakogut 4y ago
    There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
- deathanatos 4y ago
  Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)
sofixa 4y ago
Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).
rwmj 4y ago
You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.
jeroenhd 4y ago
The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.

topspin 4y ago

I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't even live migrate a firecracker VM right now. So yes, VMs are great and will ultimately resurge to the detriment of 'everything is a container' serverless orchestration, but Firecracker in its current form won't cut it. A gulf exists between giant cloud operators and their lucrative 'serverless' (only) model and everyone else. Everyone else must avoid the insurmountable costs associated with abandoning the immense value that stateful VMs provide and the minute something emerges that delivers a seamless VM orchestration system it will skyrocket.

tptacek 4y ago
Which kernel powers are you specifically talking about having been stripped away, and how are they relevant to serverside workloads? GPUs are the common one that gets brought up, but I'm not familiar with many others. Firecracker doesn't strip anything out of the guest kernel --- it boots a vanilla Linux kernel, which can be as heavy or light as you compile it to be.
- 0xbadcafebee 4y ago
  From https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/: Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). This, along with a streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB memory footprint. Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".