5 ms·

Anyone have recommendations on virtualizing windows for gaming? I'd like to keep my main desktop linux. Nvidia recently whitelisted direct gpu access, but I've

Anyone have recommendations on virtualizing windows for gaming? I'd like to keep my main desktop linux. Nvidia recently whitelisted direct gpu access, but I've been given conflicting advice as to esxi, virt manager, proxmox, etc. Surely this is a solved problem by now?

jhanschoo 6y ago
You need to have have a computer with integrated graphics / 2 graphics cards, and so can use a discrete card as a passthrough. Check out https://www.reddit.com/r/VFIO/ https://www.reddit.com/r/VFIO/
- sithadmin 6y ago
  You don’t. Pass through to a VM on ProxMox works fine without integrated graphics and a single GPU. The graphics card simply drops the physical host console after the VM begins to boot, and then displays whatever the VM is outputting.
  - sneak 6y ago
    Do you have more info about this setup?
    - sithadmin 6y ago
      I used the method described here: https://elijahliedtke.medium.com/home-lab-guides-proxmox-6-pci-e-passthrough-with-nvidia-43ccfb9424de https://elijahliedtke.medium.com/home-lab-guides-proxmox-6-p... Similar approach: https://matthewdeporter.com/proxmox/proxmox-pci-passthrough/ https://matthewdeporter.com/proxmox/proxmox-pci-passthrough/
  - jhanschoo 6y ago
    You're right, you can do passthrough with a single GPU, and there are multiple ways to do that. But note a huge caveat: you can't run an X server, whereas being able to simultaneously run desktop apps in the host and VM is a huge reason many run Windows in a VM rather than dual-boot into it.
    - deleted 6y ago
      [deleted]
    - sithadmin 6y ago
      Sure, it doesn't work for that use case, but most folks aren't going to be running a ProxMox system in a desktop use case anyway.
- deleted 6y ago
  [deleted]
- archi42 6y ago
  As of a few days ago, that info is outdated even for us consumers ;-) Technically one GPU will do, but it needs vGPU support to pass through slices of the GPU to the VM - which usually means expensive data center grade GPU (e.g. Tesla), because that feature is artificially limited by the driver. However, there is some pretty awesome, cutting edge work done in that regard: Some great people very recently released a method for unlocking the vGPU stuff on consumer grade nVidia GPUs. When that was posted on HN[0], it only supported some 10xx GPUs; but with a merge from only 13h ago, most {9,10,20,30}xx should now work. [0] https://news.ycombinator.com/item?id=26754351 https://news.ycombinator.com/item?id=26754351
  - selfhoster11 6y ago
    Thank you for this information.
byhemechi 6y ago
in my very limited testing, QEMU/KVM had the least overhead.
squarefoot 6y ago
Although they're different use cases, I'm not sure if virtualizing would be the right choice for performance issues; I've successfully run audio+MIDI software under WINE using PlayOnLinux [0] which sort of [1] sandboxes each program into a virtual drive image therefore leaving untouched your ~/.wine directory, also allowing the use of different WINE versions for each program installed if needed. I also attempted to use virtualized Windows for MIDI software but the results were disappointing: although the software worked, it was unusable due to buffer underruns everywhere and huge latencies compared to either plain WINE or PlayOnLinux. 0: https://www.playonlinux.com/en/ https://www.playonlinux.com/en/ 1: It isolates the Windows program to a certain point, but I have no idea if it's technically a sandbox built for security which would protect the system against a Windows program willing to do any damage to or data exfiltration from the home directory.
w0utert 6y ago
I do this with QEMU/KVM with passthrough of an RTX 3090, an NVME SSD, and one of the onboard USB controllers. Works like a charm, though the VM boot time is very high if you allocate a lot of RAM to it (there's some kind of bottleneck in the linux kernel when pinning huge amounts of consecutive memory pages while using passthrough, don't fully understand it but it's a known problem). Performance is indistinguishable from native, e.g. I can easily drive the screen at the max 144 Hz refresh rate, G-sync works, etc. I did put some effort in figuring out how to pin CPU threads to cores, optimize for the CPU core topology so Windows only gets cores on the same CCX (it's an AMD Zen 3, before that it was Zen 2), etc. But all of this is documented in many places. Do note that depending on your motherboard not all of this is possible if the chipset & BIOS do not provide enough MMIO isolation, you might not be able to isolate a USB controller or a separate NVME drive, or you might get only half the PCIe lanes for the passthrough GPU if you need to use the other full-lane slot for something like a second GPU.
- simfoo 6y ago
  I would pay good money for a ready to use distribution/setup that supports this incl. making sure it continues to work after updates
  - w0utert 6y ago
    The main problem would be to have to maintain the myriad of known working configurations, since they are all different depending on motherboard, BIOS, GPU, CPU, etc. If you are careful about picking the right parts (and assembling them properly), it's actually really easy to configure on any recent Linux distro, you can just click together the VM using virt-manager if you don't need anything special. I know I had to jump through a lot of hoops to make it work though, my X470 motherboard didn't isolate the USB controller without a BIOS update for example, and after that the USB controller exhibited USB FLR (function level reset) problems causing it to hang the VM. This required blacklisting it's PCI ID from the Linux kernel and patching the kernel to disable FLR (fortunately these changes were later merged into the mainline kernel). I also had problems with the second GPU, if I plugged it into any slot other than the bottom x1 slot, the motherboard BIOS would reshuffle the IOMMU groups making it impossible to pass through the NVME and USB controller, or (if I put it in the second x16 slot) it would halve the PCIe bandwidth to the RTX3080. All in all it took me the better part of a weekend to get everything working, but if I had to do it again from scratch and did some research into (in particular) the motherboard and BIOS, I would be able to set it everything up again in less than an hour.
schaefer 6y ago
I brought up a system with Xen and two video cards. Though I would want to upgrade my hardware (ha, as if anyone can buy chips right now) before I made a setup like that my daily driver.
fallenhitokiri 6y ago
Take a look at the Linus Tech Tips YouTube channel. They have multiple videos about virtualizing Windows and passing GPUs to the VMs. IIRC they’ve used and benchmarked this setup for gaming and video editing. Quick search brought up the one for video editing[1] [1] https://www.youtube.com/watch?v=1yFQd4MaKK0 https://www.youtube.com/watch?v=1yFQd4MaKK0
MinorTom 6y ago
I do this with ProxMox and a passed through AMD RX580 and it's working fine, but the disk performance is terrible, even worse than I'd expect for the hard drive it's on. Luckily that doesn't matter much for gaming. The Proxmox wiki has great resources on GPU passthrough, but you may have to install a kernel mod for the AMD gpu not to take the host down when the guest powers off. I haven't been able to do this with VMware ESXi (I've tried); I forgot if it worked in unRaid or not (But it should, in theory).
- skykooler 6y ago
  Is it possible to pass though a hard drive directly, like you do with the GPU?
  - mkesper 6y ago
    Yes. If it's not used by the host at all.
  - MinorTom 6y ago
    Not with Proxmox, no. You can use a PCIe-Sata card and pass that through if you have the slots available.
smileybarry 6y ago
If you're okay with dropping vendor-unique features (e.g.: RTX) then I've found VMware's GPU virtualization to be very good, at least in my testing of Windows-on-Windows via VMware Workstation. Looks like Workstation 16 added DX11 support on Linux hosts, too. Just make sure you go into fullscreen + (what they call) "exclusive mode", and maybe even increase the VM's priority when input is grabbed. That fixed any microstutters I had on my older CPU.

jhanschoo 6y ago

You need to have have a computer with integrated graphics / 2 graphics cards, and so can use a discrete card as a passthrough. Check out https://www.reddit.com/r/VFIO/ https://www.reddit.com/r/VFIO/

sithadmin 6y ago
You don’t. Pass through to a VM on ProxMox works fine without integrated graphics and a single GPU. The graphics card simply drops the physical host console after the VM begins to boot, and then displays whatever the VM is outputting.
- sneak 6y ago
  Do you have more info about this setup?
  - sithadmin 6y ago
    I used the method described here: https://elijahliedtke.medium.com/home-lab-guides-proxmox-6-pci-e-passthrough-with-nvidia-43ccfb9424de https://elijahliedtke.medium.com/home-lab-guides-proxmox-6-p... Similar approach: https://matthewdeporter.com/proxmox/proxmox-pci-passthrough/ https://matthewdeporter.com/proxmox/proxmox-pci-passthrough/
- jhanschoo 6y ago
  You're right, you can do passthrough with a single GPU, and there are multiple ways to do that. But note a huge caveat: you can't run an X server, whereas being able to simultaneously run desktop apps in the host and VM is a huge reason many run Windows in a VM rather than dual-boot into it.
  - deleted 6y ago
    [deleted]
  - sithadmin 6y ago
    Sure, it doesn't work for that use case, but most folks aren't going to be running a ProxMox system in a desktop use case anyway.
deleted 6y ago
[deleted]
archi42 6y ago
As of a few days ago, that info is outdated even for us consumers ;-) Technically one GPU will do, but it needs vGPU support to pass through slices of the GPU to the VM - which usually means expensive data center grade GPU (e.g. Tesla), because that feature is artificially limited by the driver. However, there is some pretty awesome, cutting edge work done in that regard: Some great people very recently released a method for unlocking the vGPU stuff on consumer grade nVidia GPUs. When that was posted on HN[0], it only supported some 10xx GPUs; but with a merge from only 13h ago, most {9,10,20,30}xx should now work. [0] https://news.ycombinator.com/item?id=26754351 https://news.ycombinator.com/item?id=26754351
- selfhoster11 6y ago
  Thank you for this information.

byhemechi 6y ago

in my very limited testing, QEMU/KVM had the least overhead.

squarefoot 6y ago

Although they're different use cases, I'm not sure if virtualizing would be the right choice for performance issues; I've successfully run audio+MIDI software under WINE using PlayOnLinux [0] which sort of [1] sandboxes each program into a virtual drive image therefore leaving untouched your ~/.wine directory, also allowing the use of different WINE versions for each program installed if needed. I also attempted to use virtualized Windows for MIDI software but the results were disappointing: although the software worked, it was unusable due to buffer underruns everywhere and huge latencies compared to either plain WINE or PlayOnLinux. 0: https://www.playonlinux.com/en/ https://www.playonlinux.com/en/ 1: It isolates the Windows program to a certain point, but I have no idea if it's technically a sandbox built for security which would protect the system against a Windows program willing to do any damage to or data exfiltration from the home directory.

w0utert 6y ago

I do this with QEMU/KVM with passthrough of an RTX 3090, an NVME SSD, and one of the onboard USB controllers. Works like a charm, though the VM boot time is very high if you allocate a lot of RAM to it (there's some kind of bottleneck in the linux kernel when pinning huge amounts of consecutive memory pages while using passthrough, don't fully understand it but it's a known problem). Performance is indistinguishable from native, e.g. I can easily drive the screen at the max 144 Hz refresh rate, G-sync works, etc. I did put some effort in figuring out how to pin CPU threads to cores, optimize for the CPU core topology so Windows only gets cores on the same CCX (it's an AMD Zen 3, before that it was Zen 2), etc. But all of this is documented in many places. Do note that depending on your motherboard not all of this is possible if the chipset & BIOS do not provide enough MMIO isolation, you might not be able to isolate a USB controller or a separate NVME drive, or you might get only half the PCIe lanes for the passthrough GPU if you need to use the other full-lane slot for something like a second GPU.

simfoo 6y ago
I would pay good money for a ready to use distribution/setup that supports this incl. making sure it continues to work after updates
- w0utert 6y ago
  The main problem would be to have to maintain the myriad of known working configurations, since they are all different depending on motherboard, BIOS, GPU, CPU, etc. If you are careful about picking the right parts (and assembling them properly), it's actually really easy to configure on any recent Linux distro, you can just click together the VM using virt-manager if you don't need anything special. I know I had to jump through a lot of hoops to make it work though, my X470 motherboard didn't isolate the USB controller without a BIOS update for example, and after that the USB controller exhibited USB FLR (function level reset) problems causing it to hang the VM. This required blacklisting it's PCI ID from the Linux kernel and patching the kernel to disable FLR (fortunately these changes were later merged into the mainline kernel). I also had problems with the second GPU, if I plugged it into any slot other than the bottom x1 slot, the motherboard BIOS would reshuffle the IOMMU groups making it impossible to pass through the NVME and USB controller, or (if I put it in the second x16 slot) it would halve the PCIe bandwidth to the RTX3080. All in all it took me the better part of a weekend to get everything working, but if I had to do it again from scratch and did some research into (in particular) the motherboard and BIOS, I would be able to set it everything up again in less than an hour.

schaefer 6y ago

I brought up a system with Xen and two video cards. Though I would want to upgrade my hardware (ha, as if anyone can buy chips right now) before I made a setup like that my daily driver.

fallenhitokiri 6y ago

Take a look at the Linus Tech Tips YouTube channel. They have multiple videos about virtualizing Windows and passing GPUs to the VMs. IIRC they’ve used and benchmarked this setup for gaming and video editing. Quick search brought up the one for video editing[1] [1] https://www.youtube.com/watch?v=1yFQd4MaKK0 https://www.youtube.com/watch?v=1yFQd4MaKK0

MinorTom 6y ago

I do this with ProxMox and a passed through AMD RX580 and it's working fine, but the disk performance is terrible, even worse than I'd expect for the hard drive it's on. Luckily that doesn't matter much for gaming. The Proxmox wiki has great resources on GPU passthrough, but you may have to install a kernel mod for the AMD gpu not to take the host down when the guest powers off. I haven't been able to do this with VMware ESXi (I've tried); I forgot if it worked in unRaid or not (But it should, in theory).

skykooler 6y ago
Is it possible to pass though a hard drive directly, like you do with the GPU?
- mkesper 6y ago
  Yes. If it's not used by the host at all.
- MinorTom 6y ago
  Not with Proxmox, no. You can use a PCIe-Sata card and pass that through if you have the slots available.

smileybarry 6y ago

If you're okay with dropping vendor-unique features (e.g.: RTX) then I've found VMware's GPU virtualization to be very good, at least in my testing of Windows-on-Windows via VMware Workstation. Looks like Workstation 16 added DX11 support on Linux hosts, too. Just make sure you go into fullscreen + (what they call) "exclusive mode", and maybe even increase the VM's priority when input is grabbed. That fixed any microstutters I had on my older CPU.