12 ms·

If you want to virtualize something with good performance on desktop windows you use Hyper-V; if you want to do it on mac you use Apple's Virtualization Framewo

If you want to virtualize something with good performance on desktop windows you use Hyper-V; if you want to do it on mac you use Apple's Virtualization Framework; if you want to do it on Linux you use KVM. Desktop virtualization products used to bring the secret sauce with them; now that every OS ships with a well-integrated and well supported type 1 hypervisor they have lost much of the reason for existing. There's only so much UI you can put in front of off-the-shelf os features and still charge hundreds of dollars per year for.

rcarmo 2y ago
They still need to. You are glossing over the fact that you need to provide device access, USB access, graphics, and a lot of things that are not necessarily provided by the "native" hypervisor (HyperKit does not do even half of what Parallels does, for instance).
- zten 2y ago
  Yeah, if you care about 3D acceleration on a Windows guest and aren't doing pcie passthrough, then KVM sure isn't going to do it. There is a driver in the works, but it's not there yet. edit: I made a mistake and got confused in my head with qemu and the lack of paravirtualized support. (It does have a PV 3D linux driver, though)
  - xxpor 2y ago
    Do any of the commercial hypervisors do that today?
    - reanimus 2y ago
      Pretty much all of them do, though the platform support varies by hypervisor/guest OS. Paravirtualized (aka non-passthrough) 3D acceleration has been implemented for well over a decade.
      - wongarsu 2y ago
        However NVIDIA limits it to datacenter GPUs. And you might need an additional license, not sure about that. In their view it's a product for Citrix and other virtual desktops, not something a normal consumer needs.
        smileybarry 2y ago
        Yes and no; you can use GPU partitioning in Hyper-V with consumer cards and Windows 10/11 client on both sides, it’s just annoying to set up, and even then there’s hoops to jump through to get decent performance. If you don’t need vendor-specific features/drivers, then VMware Workstation (even with Hyper-V enabled) supports proper guest 3D acceleration with some light GPU virtualization, up to DX11 IIRC. It doesn’t see the host’s NVIDIA/AMD/Intel card and doesn’t use that vendor’s drivers, so there’s no datacenter SKU restrictions. (But you are limited to pure DX11 & OpenGL usage, no CUDA etc.)
  - gorkish 2y ago
    KVM will happily work with real virtual GPU support from every vendor; it's the vendors (except for intel) that feel the need to artificially limit who is allowed to use these features.
    - zten 2y ago
      I was mostly hoping qemu would get paravirtualized support some day, because it is leagues ahead of VMware Player in speed. Everyone's hopes are riding on https://github.com/virtio-win/kvm-guest-drivers-windows/pull/943 https://github.com/virtio-win/kvm-guest-drivers-windows/pull....
      - gorkish 2y ago
        I guess my comments make it sound like I don't appreciate this type of work; I absolutely do. An old friend of mine[1] was responsible for the first 3d support in the vmware svga driver, so this is a space I have been following for literally decades at this point. I just think it should be the objective of vendors to offer actual GPU virtualization first and to support paravirtualization as an optimization in the cases where it is useful or superior and the tradeoffs are acceptable. [1] https://scanlime.org/ https://scanlime.org/
  - AshamedCaptain 2y ago
    There has been a driver "in the works" for the past decade. Never coming. MS/Apple do not make it easy anyway.
- jhiesey 2y ago
  Agreed. Virtualized 3d acceleration in particular still has quite a bit of "secret sauce" left in it.
  - gorkish 2y ago
    Today this is mostly implemented by having a guest driver pass calls through to a layer on the host that does the actual rendering. While I agree that there is a lot of magic to making such an arrangement work, it's a terrible awful idea to suggest that relying on a vendor's emulation layer is how things should be done today. Proper GPU virtualization and/or partitioning is the right way to do it and the vendors need to get their heads out of their ass and stop restricting its use on consumer hardware. Intel already does; you can use GVT-g to get guest gpu on any platform that wants to implement it.
    - AshamedCaptain 2y ago
      So you say having a decoupled arrangement in software (which happens to be a de facto open standard) is a "terrible awful idea" and that instead you should just rely on whatever your proprietary hardware graphics vendor proposes to you? Why? And that's assuming they propose anything at all. Even GVT-g breaks every other Linux release, is at risk of being abandoned by Intel (e.g. how they already abandoned the Xen version) or limited to specific CPU market segments, and already has ridiculous limitations such as a limit on the number of concurrent framebuffers AND framebuffer sizes (why? VMware Workstation offers you an infinitely resizable window, does it with 3D acceleration just fine, and I have never been able to tell if they have a limit on the number of simultaneous VMs... ). In the meanwhile "software-based GPU virtualization" allows me to share GPUs in the host that will never have hardware-based partitioning support (e.g. ANY consumer AMD card), and allows guests to have working 3D by implementing only one interface (e.g. https://github.com/JHRobotics/softgpu https://github.com/JHRobotics/softgpu for retro Windows) instead of having to implement drivers for every GPU in existence.
      - derefr 2y ago
        > So you say having a decoupled arrangement in software (which happens to be a de facto open standard) is a "terrible awful idea" and that instead you should just rely on whatever your proprietary hardware graphics vendor proposes to you? Why? Sandboxing, and resource quotas / allocations / reservations. By itself, a paravirtualized GPU just treats each userland workload launched by any given guest onto the GPU, as all being siblings — exactly as if there was no virtualization and you were just running multiple workloads on one host. And so, just like multiple GPU-using apps on a single non-virtualized host, these workloads will get "thin-provisioned" the resources they need, as they ask for them, with no advance reservation; and workloads may very well end up fighting over those resources, if they attempt to use a lot of them. You're just not supposed to run two things that attempt to use "as much VRAM as possible" at once. This means that, on a multi-tenant hypervisor host (e.g. the "with GPU" compute machines in most clouds), a paravirtualized GPU would give no protection at all from one tenant using all of a host GPU's resources, leaving none left over for the other guests sharing that host GPU. The cloud vendor would have guaranteed each tenant so much GPU capacity — but that guarantee would be empty! To enforce multi-tenant QoS, you need hardware-supported virtualization — i.e. the ability to make "all of the GPU" actually mean "some of the GPU", defining how much GPU that is on a per-guest basis. (And even in PC use-cases, you don't want a guest to be able to starve the host! Especially if you might be running untrusted workloads inside the guest, for e.g. forensic analysis!)
- gorkish 2y ago
  > have lost much of the reason I didn't say they have no reason to exist. I indicated they are moving towards becoming UI shells around standard OS features and/or other commodity software, which they are. Look at UTM, for instance. Even VMware Workstation and VirtualBox on Windows use HyperV under the hood if you have HyperV or WSL features enabled. While everyone still seems to be busy disagreeing with me because of <insert favorite feature>, I'll mention that HyperV does have official support for transparent GPU paravirtualization with nvidia cards, and there are plenty of other open projects in the works that strive to "bleed through" graphics/gpu/other hardware acceleration api's from host to guest on other platforms and hypervisors. With vendors finally settling around virtio as somewhat of a 'standard pipe' for this, expect rapid progress to continue.
  - AshamedCaptain 2y ago
    > Even VMware Workstation and VirtualBox on Windows use HyperV under the hood if you have HyperV or WSL features enabled. VirtualBox is consistently (and significantly) slower when it uses HyperV as backend than when it uses its original driver, and many features are not supported at all with HyperV. In fact the GUI actually shows a "tortoise" icon in the status bar when running with HyperV backend.
    - Peanuts99 2y ago
      What is the use case for using Virtualbox with Hyper-V? Why not just use Hyper-V directly?
      - AshamedCaptain 2y ago
        For a start, the list of operating systems Hyper-V supports is an order of magnitude less than what VirtualBox supports. Likewise for emulated hardware, like 3D as mentioned a number of times here. The GUI is also much better on VirtualBox. And Windows many times forces HyperV onto you, taking exclusive control of the CPU's virtualization features, thereby forcing VirtualBox to either use Hyper-V as a (terrible) backend .... or not run at all.
      - 2y ago
basil-rash 2y ago
It’s not the UI you charge for, it’s the Enterprise Support Plan.
fifteen1506 2y ago
Worse, it's actively worse VMware on Windows vs Hyper-V.
packetlost 2y ago
VMWork workstation still has a massive leg up in 3d (and to some extent, 2d) video acceleration. Many programs need this to run smoothly these days
Thaxll 2y ago
Hyper v doesn't have 3d acceleration, so if you play game or want to use linux desktops it's pretty bad.
- PaulHoule 2y ago
  WSL2 seems to virtualize the GPU pretty well, I had an easier time getting my GPU to work for machine learning inside WSL2 than I have with plain Windows and Linux in the past.
  - smileybarry 2y ago
    It does, but it’s a whole rabbit hole of specialized settings from what I can tell. Toying around with GPU-PV in Hyper-V with a Windows 11 guest was complicated and ultimately had performance & compatibility problems. (With my previous PC even deadlocking when I used the onboard video encoder from within the VM)
doublerabbit 2y ago
And if you want to do it on FreeBSD you use bHyve.
netrap 2y ago
If you want to pass through USB, SCSI, or something like that then VMWare Workstation is better than Hyper-V for sure.
NikolaNovak 2y ago
Am I the only one who explicitly does not want type 1 hyper visor on my desktop? Am I out dated? I like workstation and virtualbox because they're controllable and minimally impactful when I'm not using them. Installing hyper v (and historically even WSL - not sure if it's still the case but it was never sufficiently explicit) now makes my primary OS a guest, with potential impact on my gaming, multimedia, and other performance (and occasional flaky issues with drivers and whatnots). Am I the only grouchy geezer here?:-)
- Kwpolska 2y ago
  Apart from Hyper-V and WSL, some Windows 11 security features also depend on virtualization. Did you measure the performance hit? How often did you encounter driver trouble?
- smileybarry 2y ago
  I used to worry about this overhead too but this appears to be nothing on modern CPUs. I had minuscule differences here-and-there on Intel 9th gen (9900K) but my current Intel 13th gen (13900K) has zero performance decrease with HV enabled. (At least on any perceptible level)
  - NikolaNovak 2y ago
    Thanks - can you share your usage patterns, what kind of usage, does it include heavy gaming and media? Note, I'm less worried about percentage performance, as some things just not working well at all, because of assumptions of direct hardware access vs reality of running under hyper v. I.e. Are ALL hardware calls and capabilities 100% absolutely completely available once your main Windows install is running as a VM? Not most, majority, should be good; but actually, seamlessly all? My understanding was No, but things may have changed for the better.
ycombinator_acc 2y ago
I would never use "performance" and "Hyper-V" in the same sentence.
unnouinceput 2y ago
Hyper-V does not have PCI passthrough and with that you lost me, while ESXi does. Also I want to test my multiplatform software on all major OSes (MacOs included) ESXi is then the only one that can run Darwin in parallel with rest.
justinclift 2y ago
Snapshot support in at least Libvirt based KVM still sucks. :( As in, technically it supports snapshots. But try have a tree of different snapshots of a VM based upon different points in time. Super useful when doing integration work and testing out various approaches that evolve over time as things progress. With VMware Workstation I've been able to do that for years, for Libvirt based KVM it's not even possible. To be clear, I really wish it could be done in Libvirt/KVM too. ;)

rcarmo 2y ago

They still need to. You are glossing over the fact that you need to provide device access, USB access, graphics, and a lot of things that are not necessarily provided by the "native" hypervisor (HyperKit does not do even half of what Parallels does, for instance).

zten 2y ago
Yeah, if you care about 3D acceleration on a Windows guest and aren't doing pcie passthrough, then KVM sure isn't going to do it. There is a driver in the works, but it's not there yet. edit: I made a mistake and got confused in my head with qemu and the lack of paravirtualized support. (It does have a PV 3D linux driver, though)
- xxpor 2y ago
  Do any of the commercial hypervisors do that today?
  - reanimus 2y ago
    Pretty much all of them do, though the platform support varies by hypervisor/guest OS. Paravirtualized (aka non-passthrough) 3D acceleration has been implemented for well over a decade.
    - wongarsu 2y ago
      However NVIDIA limits it to datacenter GPUs. And you might need an additional license, not sure about that. In their view it's a product for Citrix and other virtual desktops, not something a normal consumer needs.
      - smileybarry 2y ago
        Yes and no; you can use GPU partitioning in Hyper-V with consumer cards and Windows 10/11 client on both sides, it’s just annoying to set up, and even then there’s hoops to jump through to get decent performance. If you don’t need vendor-specific features/drivers, then VMware Workstation (even with Hyper-V enabled) supports proper guest 3D acceleration with some light GPU virtualization, up to DX11 IIRC. It doesn’t see the host’s NVIDIA/AMD/Intel card and doesn’t use that vendor’s drivers, so there’s no datacenter SKU restrictions. (But you are limited to pure DX11 & OpenGL usage, no CUDA etc.)
- gorkish 2y ago
  KVM will happily work with real virtual GPU support from every vendor; it's the vendors (except for intel) that feel the need to artificially limit who is allowed to use these features.
  - zten 2y ago
    I was mostly hoping qemu would get paravirtualized support some day, because it is leagues ahead of VMware Player in speed. Everyone's hopes are riding on https://github.com/virtio-win/kvm-guest-drivers-windows/pull/943 https://github.com/virtio-win/kvm-guest-drivers-windows/pull....
    - gorkish 2y ago
      I guess my comments make it sound like I don't appreciate this type of work; I absolutely do. An old friend of mine[1] was responsible for the first 3d support in the vmware svga driver, so this is a space I have been following for literally decades at this point. I just think it should be the objective of vendors to offer actual GPU virtualization first and to support paravirtualization as an optimization in the cases where it is useful or superior and the tradeoffs are acceptable. [1] https://scanlime.org/ https://scanlime.org/
- AshamedCaptain 2y ago
  There has been a driver "in the works" for the past decade. Never coming. MS/Apple do not make it easy anyway.
jhiesey 2y ago
Agreed. Virtualized 3d acceleration in particular still has quite a bit of "secret sauce" left in it.
- gorkish 2y ago
  Today this is mostly implemented by having a guest driver pass calls through to a layer on the host that does the actual rendering. While I agree that there is a lot of magic to making such an arrangement work, it's a terrible awful idea to suggest that relying on a vendor's emulation layer is how things should be done today. Proper GPU virtualization and/or partitioning is the right way to do it and the vendors need to get their heads out of their ass and stop restricting its use on consumer hardware. Intel already does; you can use GVT-g to get guest gpu on any platform that wants to implement it.
  - AshamedCaptain 2y ago
    So you say having a decoupled arrangement in software (which happens to be a de facto open standard) is a "terrible awful idea" and that instead you should just rely on whatever your proprietary hardware graphics vendor proposes to you? Why? And that's assuming they propose anything at all. Even GVT-g breaks every other Linux release, is at risk of being abandoned by Intel (e.g. how they already abandoned the Xen version) or limited to specific CPU market segments, and already has ridiculous limitations such as a limit on the number of concurrent framebuffers AND framebuffer sizes (why? VMware Workstation offers you an infinitely resizable window, does it with 3D acceleration just fine, and I have never been able to tell if they have a limit on the number of simultaneous VMs... ). In the meanwhile "software-based GPU virtualization" allows me to share GPUs in the host that will never have hardware-based partitioning support (e.g. ANY consumer AMD card), and allows guests to have working 3D by implementing only one interface (e.g. https://github.com/JHRobotics/softgpu https://github.com/JHRobotics/softgpu for retro Windows) instead of having to implement drivers for every GPU in existence.
    - derefr 2y ago
      > So you say having a decoupled arrangement in software (which happens to be a de facto open standard) is a "terrible awful idea" and that instead you should just rely on whatever your proprietary hardware graphics vendor proposes to you? Why? Sandboxing, and resource quotas / allocations / reservations. By itself, a paravirtualized GPU just treats each userland workload launched by any given guest onto the GPU, as all being siblings — exactly as if there was no virtualization and you were just running multiple workloads on one host. And so, just like multiple GPU-using apps on a single non-virtualized host, these workloads will get "thin-provisioned" the resources they need, as they ask for them, with no advance reservation; and workloads may very well end up fighting over those resources, if they attempt to use a lot of them. You're just not supposed to run two things that attempt to use "as much VRAM as possible" at once. This means that, on a multi-tenant hypervisor host (e.g. the "with GPU" compute machines in most clouds), a paravirtualized GPU would give no protection at all from one tenant using all of a host GPU's resources, leaving none left over for the other guests sharing that host GPU. The cloud vendor would have guaranteed each tenant so much GPU capacity — but that guarantee would be empty! To enforce multi-tenant QoS, you need hardware-supported virtualization — i.e. the ability to make "all of the GPU" actually mean "some of the GPU", defining how much GPU that is on a per-guest basis. (And even in PC use-cases, you don't want a guest to be able to starve the host! Especially if you might be running untrusted workloads inside the guest, for e.g. forensic analysis!)
gorkish 2y ago
> have lost much of the reason I didn't say they have no reason to exist. I indicated they are moving towards becoming UI shells around standard OS features and/or other commodity software, which they are. Look at UTM, for instance. Even VMware Workstation and VirtualBox on Windows use HyperV under the hood if you have HyperV or WSL features enabled. While everyone still seems to be busy disagreeing with me because of <insert favorite feature>, I'll mention that HyperV does have official support for transparent GPU paravirtualization with nvidia cards, and there are plenty of other open projects in the works that strive to "bleed through" graphics/gpu/other hardware acceleration api's from host to guest on other platforms and hypervisors. With vendors finally settling around virtio as somewhat of a 'standard pipe' for this, expect rapid progress to continue.
- AshamedCaptain 2y ago
  > Even VMware Workstation and VirtualBox on Windows use HyperV under the hood if you have HyperV or WSL features enabled. VirtualBox is consistently (and significantly) slower when it uses HyperV as backend than when it uses its original driver, and many features are not supported at all with HyperV. In fact the GUI actually shows a "tortoise" icon in the status bar when running with HyperV backend.
  - Peanuts99 2y ago
    What is the use case for using Virtualbox with Hyper-V? Why not just use Hyper-V directly?
    - AshamedCaptain 2y ago
      For a start, the list of operating systems Hyper-V supports is an order of magnitude less than what VirtualBox supports. Likewise for emulated hardware, like 3D as mentioned a number of times here. The GUI is also much better on VirtualBox. And Windows many times forces HyperV onto you, taking exclusive control of the CPU's virtualization features, thereby forcing VirtualBox to either use Hyper-V as a (terrible) backend .... or not run at all.
    - 2y ago

basil-rash 2y ago

It’s not the UI you charge for, it’s the Enterprise Support Plan.

fifteen1506 2y ago

Worse, it's actively worse VMware on Windows vs Hyper-V.

packetlost 2y ago

VMWork workstation still has a massive leg up in 3d (and to some extent, 2d) video acceleration. Many programs need this to run smoothly these days

Thaxll 2y ago

Hyper v doesn't have 3d acceleration, so if you play game or want to use linux desktops it's pretty bad.

PaulHoule 2y ago
WSL2 seems to virtualize the GPU pretty well, I had an easier time getting my GPU to work for machine learning inside WSL2 than I have with plain Windows and Linux in the past.
- smileybarry 2y ago
  It does, but it’s a whole rabbit hole of specialized settings from what I can tell. Toying around with GPU-PV in Hyper-V with a Windows 11 guest was complicated and ultimately had performance & compatibility problems. (With my previous PC even deadlocking when I used the onboard video encoder from within the VM)

doublerabbit 2y ago

And if you want to do it on FreeBSD you use bHyve.

netrap 2y ago

If you want to pass through USB, SCSI, or something like that then VMWare Workstation is better than Hyper-V for sure.

NikolaNovak 2y ago

Am I the only one who explicitly does not want type 1 hyper visor on my desktop? Am I out dated? I like workstation and virtualbox because they're controllable and minimally impactful when I'm not using them. Installing hyper v (and historically even WSL - not sure if it's still the case but it was never sufficiently explicit) now makes my primary OS a guest, with potential impact on my gaming, multimedia, and other performance (and occasional flaky issues with drivers and whatnots). Am I the only grouchy geezer here?:-)

Kwpolska 2y ago
Apart from Hyper-V and WSL, some Windows 11 security features also depend on virtualization. Did you measure the performance hit? How often did you encounter driver trouble?
smileybarry 2y ago
I used to worry about this overhead too but this appears to be nothing on modern CPUs. I had minuscule differences here-and-there on Intel 9th gen (9900K) but my current Intel 13th gen (13900K) has zero performance decrease with HV enabled. (At least on any perceptible level)
- NikolaNovak 2y ago
  Thanks - can you share your usage patterns, what kind of usage, does it include heavy gaming and media? Note, I'm less worried about percentage performance, as some things just not working well at all, because of assumptions of direct hardware access vs reality of running under hyper v. I.e. Are ALL hardware calls and capabilities 100% absolutely completely available once your main Windows install is running as a VM? Not most, majority, should be good; but actually, seamlessly all? My understanding was No, but things may have changed for the better.

ycombinator_acc 2y ago

I would never use "performance" and "Hyper-V" in the same sentence.

unnouinceput 2y ago

Hyper-V does not have PCI passthrough and with that you lost me, while ESXi does. Also I want to test my multiplatform software on all major OSes (MacOs included) ESXi is then the only one that can run Darwin in parallel with rest.

justinclift 2y ago

Snapshot support in at least Libvirt based KVM still sucks. :( As in, technically it supports snapshots. But try have a tree of different snapshots of a VM based upon different points in time. Super useful when doing integration work and testing out various approaches that evolve over time as things progress. With VMware Workstation I've been able to do that for years, for Libvirt based KVM it's not even possible. To be clear, I really wish it could be done in Libvirt/KVM too. ;)