4 ms·

Create a QubesOS Gaming HVM with GPU PCI passthrough (2023)

93 points by transpute 2y ago

z_ 2y ago
Also worth mentioning Looking Glass which does a similar trick. https://looking-glass.io/ https://looking-glass.io/
- yjftsjthsd-h 2y ago
  I can't quite figure out what this is. Is it a way to run KVM VMs but make their windows show up on the host without a virtual "screen" that you have in a window?
  - Syonyk 2y ago
    My understanding, though I've not played with it, is that Looking Glass is a way to have hardware GPU accelerated guests "render in a window on the host." Normally, GPU passthrough for gaming has involved "You pass an entire GPU through to the guest, and use a separate monitor connection from that GPU for the output." So it's a VM, but not in the "interacts with the rest of the system as a first class window" sort of way. I believe Looking Glass is a collection of techniques to pass a framebuffer from the GPU accelerated guest "monitor" back to the host, so you can have full hardware GPU acceleration in a guest, and still treat it like a window. I've not messed with it, though. Hm. Though I've got enough hardware to, right now...
  - lathiat 2y ago
    What it does is uses a PCIe pass-through GPU to render the output, but instead of sending the actual video result to the actual GPU output (and thus, needing to switch input on your monitor), it renders from the GPU into a shared memory buffer which the host can display (in a window, or full screen). So it sortof works like remote desktop, but lower latency and full resolution due to using a shared memory buffer instead of TCP.
  - sureglymop 2y ago
    Yes. It's a way to have a "viewer" of the VMs screen that has as little latency as possible. My understanding is that this is achieved by configuring shared memory between host and guest. It uses some nvidia capture api/sdk to capture the screen of the guest and write it to that shared memory. It has a server component running on the guest and a client component running on the host. I've been using it for a while with a windows 11 guest. It works but I wouldn't say it's production ready as I often get crashes. Though it's usually the server/windows component that crashes and not the client. If going for a passthrough setup I'd generally recommend setting up evdev. It allows you to switch keayboard and mouse input to the vm by e.g. pressing both Ctrl keys. I then just have to change the video input on my display.
    - transpute 2y ago
      > change the video input on my display Many monitors support input switching via DDC, using CLI software that can be mapped to a hotkey with keyboard macro automation, https://joostrijneveld.nl/posts/2024-06-04-ddc-input-switching/ https://joostrijneveld.nl/posts/2024-06-04-ddc-input-switchi...
      - sureglymop 2y ago
        It's a good addition. The reason why I personally don't have this setup is ironically because of Looking Glass. Sometimes I want to switch my mouse and keyboard over but still want to just see the video output of my vm in a Looking Glass client window.
- moondev 2y ago
  Does this work for GPUs without a physical display out port? For example an Nvidia A30 or H100
  - transpute 2y ago
    Looking Glass is for graphics remoting, including GPUs without physical display outputs. It's not for GPGPU API remoting of GPUs designed for ML/AI.
- rubatuga 2y ago
  Or moonlight for that mattwr
- deleted 2y ago
  [deleted]
Syonyk 2y ago
2023, but has been updated and iterated on over time, so should be reasonably current. It is... still an awful lot of work, though. I'm glad it can be done, though I've certainly not been motivated enough to try. I just keep a different machine around for anything gaming-ish, and don't worry about it. That said, if you have skipped out on trying Qubes because you think you lack sufficient hardware, my Qubes daily driver (and primary computer for personal use) is a 2015-era Lenovo X250, with a 2C/4T I7-5600U, 16GB RAM, and a SATA SSD. It's limited in some ways, I can feel when I'm trying to do too much on it, but it is quite useful, and has been for some while. The memory balancing between AppVMs works nicely, and you can safely reduce memory allocation for a lot of VMs. A lot of my running Debian12 VMs are between 1GB and 2GB practical allocation, and work just fine.
- zvmaz 2y ago
  Hi there. I just wanted to say that I like your blog, specifically the little kitty that follows your cursor. Incredible how endearing that little animation is!
  - Syonyk 2y ago
    Neko! It's been software for many decades - https://en.wikipedia.org/wiki/Neko_(software) https://en.wikipedia.org/wiki/Neko_(software) I'm glad it's enjoyed. I very much like having him chasing around, even if it's a bit distracting at times.
jakebasile 2y ago
I appreciate that the journey might be the destination here, but you really don't need to do this anymore if the goal is to play games on Linux. Valve et al. have done so much Proton work that nearly every game I try just works. Aside from the experience, this could be useful if you want to play games that have kernel level rootkits.
- NullPrefix 2y ago
  You still need GPU passthrough if you want to use ant actual GPU instead of llvmpipe or whatever else renderer. Mind you, QubeOS is virtualization OS, Linux on QubesOS does not have access to GPU by default
- Syonyk 2y ago
  QubesOS is very much not "Linux on the hardware." It is silos of isolation, interacting freely at the window level. I wrote on it some while back: https://www.sevarg.net/2023/07/29/qubes-os-silos-of-isolation/ https://www.sevarg.net/2023/07/29/qubes-os-silos-of-isolatio... The concept is that you have a lot of isolated "AppVMs" running applications, in silos that cannot talk to each other. So, right now, I'm logged into HN with my "random-web" VM - that has nothing of interest in it beyond some PDFs I've downloaded. This, for instance, has zero access (except by exploiting and passing through Xen) my "sysadmin" VM, which contains my SSH keys for various things, and which I use for sysadmin type tasks. I've got other silos as well. All the windows from all these VMs interact like normal on my desktop - I'm not dealing with "Okay, this VM is for this, that VM is for that." Things just work smoothly and as expected. A side effect of this, though, is that the AppVMs have no hardware acceleration of anything graphical. It's all software rendering in them. So gaming is normally right out - except, this talks about how to pass another GPU through to do this sort of thing, if you want.
  - vlovich123 2y ago
    > Things just work smoothly and as expected. Do you have the ability to pass files between or drag and drop files between isolated levels? Can I mount a shared folder into multiple isolated apps so they all have access? I like QubeOS in theory but I’m hesitant to try it due to (perhaps misplaced) UX concerns.
postcert 2y ago
I'd be curios to try vGpu with Qubes. It's definitely a security issue and has been left behind on newer NV consumer hardware but would be neat for low-risk qubes. I do have to admit that the performance is still great w/o hardware acceleration.
- transpute 2y ago
  An old AMD workstation GPU supported SR-IOV partitioning, https://open-iov.org/index.php/GPU_Support#AMD https://open-iov.org/index.php/GPU_Support#AMD > It's definitely a security issue Have there been public exploits of Intel or Nvidia SR-IOV implementations, to identify where hardening is needed?
  - postcert 2y ago
    Not that I’ve seen or heard of but my assumption is that it is mainly due to the relative rarity of partial/vgpu offerings outside of organizations. (Other than Blender render farming and some smaller upstart p2p gpu time reselling.) I’d say Intel’s 12th gen igpu has “best supported” consumer vGpu offerings available and it’s through sr-ion I believe as well. I’ve had it on the back burner on my homelab until a recent performant gui need had come up.
codecraze 2y ago
A fee years ago (~2017-2018) I configured a machine with UnRAID where i had a vm running windows 10 for gaming. It worked great passed the hard configuration haha I was able to play on my windows 10 machine vm and work on a macos vm.
robcohen 2y ago
While it is possible and very cool to do PCI pass-through for GPUs, a huge problem with this approach is that online games have anti-cheat that checks to see if the game is running in a VM. So games without that mostly games that are single player will work but anything networked will likely not. While it’s absolutely possible to play the cat and mouse game where you beat the Anti-cheat engine, It’s frankly an awful lot of work for such little benefit. You’re better off just using the service like GeForce Now Or something similar for networked games.

93 points by transpute 2y ago

z_ 2y ago

Also worth mentioning Looking Glass which does a similar trick. https://looking-glass.io/ https://looking-glass.io/

yjftsjthsd-h 2y ago
I can't quite figure out what this is. Is it a way to run KVM VMs but make their windows show up on the host without a virtual "screen" that you have in a window?
- Syonyk 2y ago
  My understanding, though I've not played with it, is that Looking Glass is a way to have hardware GPU accelerated guests "render in a window on the host." Normally, GPU passthrough for gaming has involved "You pass an entire GPU through to the guest, and use a separate monitor connection from that GPU for the output." So it's a VM, but not in the "interacts with the rest of the system as a first class window" sort of way. I believe Looking Glass is a collection of techniques to pass a framebuffer from the GPU accelerated guest "monitor" back to the host, so you can have full hardware GPU acceleration in a guest, and still treat it like a window. I've not messed with it, though. Hm. Though I've got enough hardware to, right now...
- lathiat 2y ago
  What it does is uses a PCIe pass-through GPU to render the output, but instead of sending the actual video result to the actual GPU output (and thus, needing to switch input on your monitor), it renders from the GPU into a shared memory buffer which the host can display (in a window, or full screen). So it sortof works like remote desktop, but lower latency and full resolution due to using a shared memory buffer instead of TCP.
- sureglymop 2y ago
  Yes. It's a way to have a "viewer" of the VMs screen that has as little latency as possible. My understanding is that this is achieved by configuring shared memory between host and guest. It uses some nvidia capture api/sdk to capture the screen of the guest and write it to that shared memory. It has a server component running on the guest and a client component running on the host. I've been using it for a while with a windows 11 guest. It works but I wouldn't say it's production ready as I often get crashes. Though it's usually the server/windows component that crashes and not the client. If going for a passthrough setup I'd generally recommend setting up evdev. It allows you to switch keayboard and mouse input to the vm by e.g. pressing both Ctrl keys. I then just have to change the video input on my display.
  - transpute 2y ago
    > change the video input on my display Many monitors support input switching via DDC, using CLI software that can be mapped to a hotkey with keyboard macro automation, https://joostrijneveld.nl/posts/2024-06-04-ddc-input-switching/ https://joostrijneveld.nl/posts/2024-06-04-ddc-input-switchi...
    - sureglymop 2y ago
      It's a good addition. The reason why I personally don't have this setup is ironically because of Looking Glass. Sometimes I want to switch my mouse and keyboard over but still want to just see the video output of my vm in a Looking Glass client window.
moondev 2y ago
Does this work for GPUs without a physical display out port? For example an Nvidia A30 or H100
- transpute 2y ago
  Looking Glass is for graphics remoting, including GPUs without physical display outputs. It's not for GPGPU API remoting of GPUs designed for ML/AI.
rubatuga 2y ago
Or moonlight for that mattwr
deleted 2y ago
[deleted]

Syonyk 2y ago

2023, but has been updated and iterated on over time, so should be reasonably current. It is... still an awful lot of work, though. I'm glad it can be done, though I've certainly not been motivated enough to try. I just keep a different machine around for anything gaming-ish, and don't worry about it. That said, if you have skipped out on trying Qubes because you think you lack sufficient hardware, my Qubes daily driver (and primary computer for personal use) is a 2015-era Lenovo X250, with a 2C/4T I7-5600U, 16GB RAM, and a SATA SSD. It's limited in some ways, I can feel when I'm trying to do too much on it, but it is quite useful, and has been for some while. The memory balancing between AppVMs works nicely, and you can safely reduce memory allocation for a lot of VMs. A lot of my running Debian12 VMs are between 1GB and 2GB practical allocation, and work just fine.

zvmaz 2y ago
Hi there. I just wanted to say that I like your blog, specifically the little kitty that follows your cursor. Incredible how endearing that little animation is!
- Syonyk 2y ago
  Neko! It's been software for many decades - https://en.wikipedia.org/wiki/Neko_(software) https://en.wikipedia.org/wiki/Neko_(software) I'm glad it's enjoyed. I very much like having him chasing around, even if it's a bit distracting at times.

jakebasile 2y ago

I appreciate that the journey might be the destination here, but you really don't need to do this anymore if the goal is to play games on Linux. Valve et al. have done so much Proton work that nearly every game I try just works. Aside from the experience, this could be useful if you want to play games that have kernel level rootkits.

NullPrefix 2y ago
You still need GPU passthrough if you want to use ant actual GPU instead of llvmpipe or whatever else renderer. Mind you, QubeOS is virtualization OS, Linux on QubesOS does not have access to GPU by default
Syonyk 2y ago
QubesOS is very much not "Linux on the hardware." It is silos of isolation, interacting freely at the window level. I wrote on it some while back: https://www.sevarg.net/2023/07/29/qubes-os-silos-of-isolation/ https://www.sevarg.net/2023/07/29/qubes-os-silos-of-isolatio... The concept is that you have a lot of isolated "AppVMs" running applications, in silos that cannot talk to each other. So, right now, I'm logged into HN with my "random-web" VM - that has nothing of interest in it beyond some PDFs I've downloaded. This, for instance, has zero access (except by exploiting and passing through Xen) my "sysadmin" VM, which contains my SSH keys for various things, and which I use for sysadmin type tasks. I've got other silos as well. All the windows from all these VMs interact like normal on my desktop - I'm not dealing with "Okay, this VM is for this, that VM is for that." Things just work smoothly and as expected. A side effect of this, though, is that the AppVMs have no hardware acceleration of anything graphical. It's all software rendering in them. So gaming is normally right out - except, this talks about how to pass another GPU through to do this sort of thing, if you want.
- vlovich123 2y ago
  > Things just work smoothly and as expected. Do you have the ability to pass files between or drag and drop files between isolated levels? Can I mount a shared folder into multiple isolated apps so they all have access? I like QubeOS in theory but I’m hesitant to try it due to (perhaps misplaced) UX concerns.

postcert 2y ago

I'd be curios to try vGpu with Qubes. It's definitely a security issue and has been left behind on newer NV consumer hardware but would be neat for low-risk qubes. I do have to admit that the performance is still great w/o hardware acceleration.

transpute 2y ago
An old AMD workstation GPU supported SR-IOV partitioning, https://open-iov.org/index.php/GPU_Support#AMD https://open-iov.org/index.php/GPU_Support#AMD > It's definitely a security issue Have there been public exploits of Intel or Nvidia SR-IOV implementations, to identify where hardening is needed?
- postcert 2y ago
  Not that I’ve seen or heard of but my assumption is that it is mainly due to the relative rarity of partial/vgpu offerings outside of organizations. (Other than Blender render farming and some smaller upstart p2p gpu time reselling.) I’d say Intel’s 12th gen igpu has “best supported” consumer vGpu offerings available and it’s through sr-ion I believe as well. I’ve had it on the back burner on my homelab until a recent performant gui need had come up.

codecraze 2y ago

A fee years ago (~2017-2018) I configured a machine with UnRAID where i had a vm running windows 10 for gaming. It worked great passed the hard configuration haha I was able to play on my windows 10 machine vm and work on a macos vm.

robcohen 2y ago

While it is possible and very cool to do PCI pass-through for GPUs, a huge problem with this approach is that online games have anti-cheat that checks to see if the game is running in a VM. So games without that mostly games that are single player will work but anything networked will likely not. While it’s absolutely possible to play the cat and mouse game where you beat the Anti-cheat engine, It’s frankly an awful lot of work for such little benefit. You’re better off just using the service like GeForce Now Or something similar for networked games.