5 ms·

Tangential question: why does it normally take so long to start traditional VMs in the first place? At least on Windows, if you start a traditional VM, it takes

Tangential question: why does it normally take so long to start traditional VMs in the first place? At least on Windows, if you start a traditional VM, it takes several seconds for it to start running anything. Edit: when I say anything, I'm not talking user programs. I mean as in, before even the first instruction of the firmware -- before even the virtual disk file is zeroed out, in cases where it needs to be. You literally can't pause the VM during this interval because the window hasn't even popped up yet, and even when it has, you still can't for a while because it literally hasn't started running anything. So the kernel and even firmware initialization slowness are entirely irrelevant to my question. Why is that?

diggan 1y ago
I mean it is basically booting a computer from scratch, kind of makes sense. You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. I guess there is a bunch of processes for security as well, like wiping like zeroing pages and similar things that takes additional time. If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.
- dataflow 1y ago
  > You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. That's not what I'm asking. I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
  - drewg123 1y ago
    Without any context in terms of what the VM is doing or what VMM software you use, my best guess is that the OS/VMM are pre-allocating memory for the VM. This might involve paging out other processes' memory, which could take some time. I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening when the VM is started..
    - dataflow 1y ago
      VirtualBox on Windows, primarily. Though I feel like haven't seen other VMs in the past start up a whole ton faster (maybe a somewhat) (ignoring WSL2). Page files are already disabled, there's plenty of free RAM, and it makes no difference how little RAM the guest is allocated (even if it's 256MB). So no, those are not the issues. VirtualBox itself seems to be doing something slow during that time and I don't know what that is.
      - mynameisvlad 1y ago
        So the issue is pretty clearly with VirtualBox itself, but you are making it sound like it's an issue with VMs on Windows or in general.
      - gopher_space 1y ago
        I remembered something about VirtualBox not playing nicely with Hyper-V on Windows, and dug up a possibly relevant post[0] on their forums. IIRC we ended up moving a few build systems to Docker and dropping VirtualBox because of hyper-v related issues, but it's been a few years. [0] https://forums.virtualbox.org/viewtopic.php?t=112113 https://forums.virtualbox.org/viewtopic.php?t=112113
        dataflow 1y ago
        That's the unrelated green-turtle issue. It's only relevant after the guest has actually started running instructions. I'm talking about before that point.
jeroenhd 1y ago
You can optimize a lot to start a Linux kernel in under a second, but if you're using a standard kernel, there are all manners of timeouts and poll attempts that make the kernel waste time booting. There's also a non-trivial amount of time the VM spends in the UEFI/CSM system preparing the virtual hardware and initializing the system environment for your bootloader. I'm pretty sure WSL2 uses a special kernel to avoid the unnecessary overhead. You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly. There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/ On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.
- dataflow 1y ago
  That's not what I'm asking about. I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
  - hnuser123456 1y ago
    probably the intel ME setting up for virtualization in a way that it can infiltrate
    - LoganDark 1y ago
      Ah yes, the source of all slowness in the CPU: hostile backdoors taking their time to compromise the work. Classic...
orev 1y ago
I think you need to provide more details on what VM software you’re using. On VirtualBox what you describe is very noticeable, and it didn’t have that delay in older versions. So it could be just an issue with that VM software and not a general “traditional VMs” issue.
- dataflow 1y ago
  Yup I'm asking about VirtualBox mainly, I just don't understand what the heck it's doing during that time that takes so long. Although I don't recall other VMs (like say, Hyper-V) being dramatically different either (ignoring WSL2 here).
  - _factor 1y ago
    Try disabling Windows Defender and trying again.
    - dataflow 1y ago
      Are you just guessing or have you actually seen the delay I'm talking about disappear as a result of this (or as a result of anything else for that matter)? Because I've already done this (yes, entirely, even the kernel mode drivers) and it's definitely not the issue.
      - hinkley 1y ago
        There was a release of subversion back in the day that reduced the number of files that were opened during a repo action like pull, and the number of times any one file got opened. On Linux it ran about 2-3x faster. Very nice change. On windows it was almost 10x faster. On the project where this change was released, my morning ritual was to come in, log on, run an svn pull command, lock my screen and go get coffee. I had at least ten minutes to kill after I got coffee, if the pot wasn’t empty when I got there. Windows is hot garbage about fopen particularly when virus scanning is on.
  - icedchai 1y ago
speed_spread 1y ago
Creating the VM itself is fast. It depends on what you run in it. Unikernel VMs can start in a few milliseconds. For example, checkout OSv.
- dataflow 1y ago
  You're saying this is true on a Windows host?
  - akdev1l 1y ago
    Yes. The delay you’re complaining about happens because you are looking at general hypervisors which also come with virtualized hardware and need to mimic a bunch of stuff so that most software will work as usual. For example: your VM starts up with the CPU in 16 bit mode because that’s just how things work in x86 and then it waits for the guest OS to set the CPU into 64 bit mode. This is completely unnecessary if you just want to run x86-64 code in a virtualized environment and you control the guest kernel and can just assume things are in 64bit mode because it’s not the 70s or whatever The guest OS would also need to probe few ports to get a bootable disk. If you control the kernel then you can just not do that and boot directly. There’s a ton of stuff that isn’t needed
    - dataflow 1y ago
      The 16 bit mode stuff and the guest OS probes are after what I'm asking, not before.
      - akdev1l 1y ago
        No it is not. The “first instruction in the BIOS” is 16 bit mode code when dealing with an x86 VM. A virtual environment doesn’t even really need any BIOS or anything like that. You can feel free to test with qemu direct kernel booting to see this skips a lot of delay without even having to use a specialized hypervisor like firecracker
      - speed_spread 1y ago
dist-epoch 1y ago
Sounds like a VirtualBox problem. I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.
akdev1l 1y ago
The answer is that it doesn’t have to be like that. In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility. If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can: > Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.
jiggawatts 1y ago
Try Windows Server Core on an SSD. I've seen VMs launch in low single-digit seconds. You can strip it down even further by removing non-64-bit support, Defender, etc...
BobbyTables2 1y ago
In Linux, VM memory allocations can be slow if it tries to allocate GBs of RAM using 4K pages. There are ways to help it allocate 1GB at a time which vastly speeds it up. Windows probably has an equivalent.
- pdimitar 1y ago
  Is this specifically for during boot time? Also, any links?

diggan 1y ago

I mean it is basically booting a computer from scratch, kind of makes sense. You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. I guess there is a bunch of processes for security as well, like wiping like zeroing pages and similar things that takes additional time. If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.

dataflow 1y ago
> You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. That's not what I'm asking. I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
- drewg123 1y ago
  Without any context in terms of what the VM is doing or what VMM software you use, my best guess is that the OS/VMM are pre-allocating memory for the VM. This might involve paging out other processes' memory, which could take some time. I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening when the VM is started..
  - dataflow 1y ago
    VirtualBox on Windows, primarily. Though I feel like haven't seen other VMs in the past start up a whole ton faster (maybe a somewhat) (ignoring WSL2). Page files are already disabled, there's plenty of free RAM, and it makes no difference how little RAM the guest is allocated (even if it's 256MB). So no, those are not the issues. VirtualBox itself seems to be doing something slow during that time and I don't know what that is.
    - mynameisvlad 1y ago
      So the issue is pretty clearly with VirtualBox itself, but you are making it sound like it's an issue with VMs on Windows or in general.
    - gopher_space 1y ago
      I remembered something about VirtualBox not playing nicely with Hyper-V on Windows, and dug up a possibly relevant post[0] on their forums. IIRC we ended up moving a few build systems to Docker and dropping VirtualBox because of hyper-v related issues, but it's been a few years. [0] https://forums.virtualbox.org/viewtopic.php?t=112113 https://forums.virtualbox.org/viewtopic.php?t=112113
      - dataflow 1y ago
        That's the unrelated green-turtle issue. It's only relevant after the guest has actually started running instructions. I'm talking about before that point.

jeroenhd 1y ago

You can optimize a lot to start a Linux kernel in under a second, but if you're using a standard kernel, there are all manners of timeouts and poll attempts that make the kernel waste time booting. There's also a non-trivial amount of time the VM spends in the UEFI/CSM system preparing the virtual hardware and initializing the system environment for your bootloader. I'm pretty sure WSL2 uses a special kernel to avoid the unnecessary overhead. You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly. There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/ On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.

dataflow 1y ago
That's not what I'm asking about. I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
- hnuser123456 1y ago
  probably the intel ME setting up for virtualization in a way that it can infiltrate
  - LoganDark 1y ago
    Ah yes, the source of all slowness in the CPU: hostile backdoors taking their time to compromise the work. Classic...

orev 1y ago

I think you need to provide more details on what VM software you’re using. On VirtualBox what you describe is very noticeable, and it didn’t have that delay in older versions. So it could be just an issue with that VM software and not a general “traditional VMs” issue.

dataflow 1y ago
Yup I'm asking about VirtualBox mainly, I just don't understand what the heck it's doing during that time that takes so long. Although I don't recall other VMs (like say, Hyper-V) being dramatically different either (ignoring WSL2 here).
- _factor 1y ago
  Try disabling Windows Defender and trying again.
  - dataflow 1y ago
    Are you just guessing or have you actually seen the delay I'm talking about disappear as a result of this (or as a result of anything else for that matter)? Because I've already done this (yes, entirely, even the kernel mode drivers) and it's definitely not the issue.
    - hinkley 1y ago
      There was a release of subversion back in the day that reduced the number of files that were opened during a repo action like pull, and the number of times any one file got opened. On Linux it ran about 2-3x faster. Very nice change. On windows it was almost 10x faster. On the project where this change was released, my morning ritual was to come in, log on, run an svn pull command, lock my screen and go get coffee. I had at least ten minutes to kill after I got coffee, if the pot wasn’t empty when I got there. Windows is hot garbage about fopen particularly when virus scanning is on.
- icedchai 1y ago

speed_spread 1y ago

Creating the VM itself is fast. It depends on what you run in it. Unikernel VMs can start in a few milliseconds. For example, checkout OSv.

dataflow 1y ago
You're saying this is true on a Windows host?
- akdev1l 1y ago
  Yes. The delay you’re complaining about happens because you are looking at general hypervisors which also come with virtualized hardware and need to mimic a bunch of stuff so that most software will work as usual. For example: your VM starts up with the CPU in 16 bit mode because that’s just how things work in x86 and then it waits for the guest OS to set the CPU into 64 bit mode. This is completely unnecessary if you just want to run x86-64 code in a virtualized environment and you control the guest kernel and can just assume things are in 64bit mode because it’s not the 70s or whatever The guest OS would also need to probe few ports to get a bootable disk. If you control the kernel then you can just not do that and boot directly. There’s a ton of stuff that isn’t needed
  - dataflow 1y ago
    The 16 bit mode stuff and the guest OS probes are after what I'm asking, not before.
    - akdev1l 1y ago
      No it is not. The “first instruction in the BIOS” is 16 bit mode code when dealing with an x86 VM. A virtual environment doesn’t even really need any BIOS or anything like that. You can feel free to test with qemu direct kernel booting to see this skips a lot of delay without even having to use a specialized hypervisor like firecracker
    - speed_spread 1y ago

dist-epoch 1y ago

Sounds like a VirtualBox problem. I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.

akdev1l 1y ago

The answer is that it doesn’t have to be like that. In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility. If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can: > Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.

jiggawatts 1y ago

Try Windows Server Core on an SSD. I've seen VMs launch in low single-digit seconds. You can strip it down even further by removing non-64-bit support, Defender, etc...

BobbyTables2 1y ago

In Linux, VM memory allocations can be slow if it tries to allocate GBs of RAM using 4K pages. There are ways to help it allocate 1GB at a time which vastly speeds it up. Windows probably has an equivalent.

pdimitar 1y ago
Is this specifically for during boot time? Also, any links?