3 ms·

What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I fe

What I want to know is how I read all these posts about starting VMs in <3 ms and I'm still waiting 15 minutes for VMs? … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.

russell_h 4y ago
Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
- easton 4y ago
  15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
  - jakogut 4y ago
    There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
- deathanatos 4y ago
  Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)
sofixa 4y ago
Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).
rwmj 4y ago
You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.
jeroenhd 4y ago
The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.

russell_h 4y ago

Wow, what OS are you booting? I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes. If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.

easton 4y ago
15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot. You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).
- jakogut 4y ago
  There's your problem. I can boot a tinyconfig Linux kernel that's 600K in QEMU using the microvm machine type in milliseconds. A basic Windows install is tens of gigabytes, and doesn't have the capability of being stripped down to the barebones essentials.
deathanatos 4y ago
Ubuntu. It's not the OS. (Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)

sofixa 4y ago

Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier. Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape. Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).

rwmj 4y ago

You should be able to provision, customize and boot a full VM in 60 seconds or so. Maybe 2-3 minutes if it involves copying in a lot of data and SELinux relabelling. If it's taking too much longer than that it's something wrong with the cloud or environment you're using.

jeroenhd 4y ago

The Big Cloud Providers (Amazon, Microsoft, Google) are all terribly slow. This is probably because of the (administrative) overhead of provisioning a VM to a customer, setting up billing, any private networking you may need, redundant storage in three timezones, and probably tons of scripts and hooks I don't even know about. In contrast, Firecracker restores a VM state to do session resumption. The OS has practically already been booted the moment execution for your specific VM begins. That said, creating a brand new VM takes less than a second on my machine and then maybe 30 seconds for a full first boot, surely the super smart people at these huge cloud companies can figure out a way to get up and running in less than a minute. The ability to spawn quick disposable VMs is one of the reasons I stick to local VMs for dev work rather than some expensive cloud solution.