4 ms·

I really hate when "USB Passthrough" is used in situations when, at best, a "USB over ethernet proxy" is what is happening. That's not passthrough... It introdu

by Izmaki 2y ago

I really hate when "USB Passthrough" is used in situations when, at best, a "USB over ethernet proxy" is what is happening. That's not passthrough... It introduces a whole range of disadvantages that regular passthrough does not (and advanced passthrough might not) have.

arghwhat 2y ago
Eh? QEMU USB passthrough is true USB passthrough. The problems with USB passthrough stem from issues related to USB controllers themselves and how device enumeration works, with the only better solution being PCIe passthrough of entire USB controllers... Which then present a different set of problems. Speaking from experience in large VM test farms with a significant amount of forwarded hardware. (However, "USB over ethernet proxy" is also a true passthrough, just one with higher latency than VirtIO.)
- ThatMedicIsASpy 2y ago
  On proxmox with a USB DAC/AMP it is impossible to get correct audio without pcie passthrough of the usb controller
  - Izmaki 2y ago
    Yeah, isochronous mode is unfortunately not supported for USB passthrough on Proxmox. There were experimental implementations in oVirt back in the days (that is: experimental implementations in a non-prod, only-for-evaluation solution...).
  - m463 2y ago
    I've run into this too. And I had to do a bunch of investigation into iommu groups to find a controller that wasn't shared.
- Izmaki 2y ago
  I skimmed the README only and just saw the big section of USB over ethernet with the video image and everything, not the tiny mentioning of VFIO above it. Lol. But tell me please, which problems do you have with PCIe passthrough? Also speaking from experience in large VM test farms with a significant amount of forwarded hardware. I've never experienced problems with hundreds of machines doing exactly this, for years.
  - arghwhat 2y ago
    PCIe passthrough has a few quirks: 1. VMs operate on a copy of certain PCIe descriptors obtained during enumeration/when forwarding was setup, meaning that some firwmare updates that depend on these changing cannot work correctly. The exact details have left my memory. 2. Foo states that only happen when forwarding. Hardware that seems so stable when used directly that bugs would seem inconceivable enter into broken states when forwarded and fail to initialize within the VM. Hardware and drivers are both full of bugs, and things become "fun" when either get surprised. You can deal with it when you're doing the forwarding your own hardware and using your own drivers so discovered issues can be debugged and sorted out, but it's much less fun when you're forwarding stuff from other vendors out of necessity. Dealt with this one just this morning. 3. Reset bugs. Hardware reset and sequencing is a tricky area (speaking from old FPGA experience), and some devices cannot recover without a full power cycle. In some cases, I can recover the device by stopping the forward, removing the device (echo 1 > /sys/bus/pci/devices/.../remove), rescanning and letting the host kernel temporarily load drivers and initialize the device, and then forward it again. Did that today. 4. Host crashes. Yay. Forwarding a single device on a user machine that still gets regular reboots tends to work fine, but things get hairy when you scale this up. I've had to do a lot of automation of things like handing devices back to the hypervisor for recovery and firmware management.
    - Izmaki 2y ago
      Strange... Sounds like you may be doing too many things manually or that what you're testing is the device that is connected directly to USB? In my case I need 3rd party USB devices (that always just work(™)) to communicate and interact with hardware. Been automating/running literally hundreds of these configurations without a single issue related to USB or PCI passthrough. Even got switchable HUBs for USB in the mix sometimes, too (for power cycling specific USB devices). Works fine as well.
      - arghwhat 2y ago
        "Manually"? There is only QEMU/KVM, how many layers you put in between does not matter. Proxmox is just a pile of perl scripts doing the same. My experience is in testing both USB downstream devices and PCIe devices developed in-house. Some of the forwarded devices might be 3rd-party devices like hubs, relays for power cycling and USB isolators to simulate hot-plug, but the DUTs are stuff we manufacture. In the USB test scenarios (we have about ~100 such machines, on average connected to a dozen DUTs, some more), the symptom of failure is generally that the entire controller can discover downstream devices but permanently fail to communicate with any of them, or that the controller itself fails to initialize entirely. The PCIe test scenarios is not something I actively work with anymore, but involves a server room full of machines with 4-7 DUTs each and much more custom handling - such as hot-unplugging the device from the VM, resetting and firmware updating the device, and hot-plugging it back as part of the test running in that VM - as testing PCIe devices themselves exercise many more issues that you don't see with standardized hardware. I have done this for about a decade, so I've been through a few iterations and tech stacks. One can find things that work, but it's not in any way or form guaranteed to work.

by Izmaki 2y ago

arghwhat 2y ago

Eh? QEMU USB passthrough is true USB passthrough. The problems with USB passthrough stem from issues related to USB controllers themselves and how device enumeration works, with the only better solution being PCIe passthrough of entire USB controllers... Which then present a different set of problems. Speaking from experience in large VM test farms with a significant amount of forwarded hardware. (However, "USB over ethernet proxy" is also a true passthrough, just one with higher latency than VirtIO.)

ThatMedicIsASpy 2y ago
On proxmox with a USB DAC/AMP it is impossible to get correct audio without pcie passthrough of the usb controller
- Izmaki 2y ago
  Yeah, isochronous mode is unfortunately not supported for USB passthrough on Proxmox. There were experimental implementations in oVirt back in the days (that is: experimental implementations in a non-prod, only-for-evaluation solution...).
- m463 2y ago
  I've run into this too. And I had to do a bunch of investigation into iommu groups to find a controller that wasn't shared.
Izmaki 2y ago
I skimmed the README only and just saw the big section of USB over ethernet with the video image and everything, not the tiny mentioning of VFIO above it. Lol. But tell me please, which problems do you have with PCIe passthrough? Also speaking from experience in large VM test farms with a significant amount of forwarded hardware. I've never experienced problems with hundreds of machines doing exactly this, for years.
- arghwhat 2y ago
  PCIe passthrough has a few quirks: 1. VMs operate on a copy of certain PCIe descriptors obtained during enumeration/when forwarding was setup, meaning that some firwmare updates that depend on these changing cannot work correctly. The exact details have left my memory. 2. Foo states that only happen when forwarding. Hardware that seems so stable when used directly that bugs would seem inconceivable enter into broken states when forwarded and fail to initialize within the VM. Hardware and drivers are both full of bugs, and things become "fun" when either get surprised. You can deal with it when you're doing the forwarding your own hardware and using your own drivers so discovered issues can be debugged and sorted out, but it's much less fun when you're forwarding stuff from other vendors out of necessity. Dealt with this one just this morning. 3. Reset bugs. Hardware reset and sequencing is a tricky area (speaking from old FPGA experience), and some devices cannot recover without a full power cycle. In some cases, I can recover the device by stopping the forward, removing the device (echo 1 > /sys/bus/pci/devices/.../remove), rescanning and letting the host kernel temporarily load drivers and initialize the device, and then forward it again. Did that today. 4. Host crashes. Yay. Forwarding a single device on a user machine that still gets regular reboots tends to work fine, but things get hairy when you scale this up. I've had to do a lot of automation of things like handing devices back to the hypervisor for recovery and firmware management.
  - Izmaki 2y ago
    Strange... Sounds like you may be doing too many things manually or that what you're testing is the device that is connected directly to USB? In my case I need 3rd party USB devices (that always just work(™)) to communicate and interact with hardware. Been automating/running literally hundreds of these configurations without a single issue related to USB or PCI passthrough. Even got switchable HUBs for USB in the mix sometimes, too (for power cycling specific USB devices). Works fine as well.
    - arghwhat 2y ago
      "Manually"? There is only QEMU/KVM, how many layers you put in between does not matter. Proxmox is just a pile of perl scripts doing the same. My experience is in testing both USB downstream devices and PCIe devices developed in-house. Some of the forwarded devices might be 3rd-party devices like hubs, relays for power cycling and USB isolators to simulate hot-plug, but the DUTs are stuff we manufacture. In the USB test scenarios (we have about ~100 such machines, on average connected to a dozen DUTs, some more), the symptom of failure is generally that the entire controller can discover downstream devices but permanently fail to communicate with any of them, or that the controller itself fails to initialize entirely. The PCIe test scenarios is not something I actively work with anymore, but involves a server room full of machines with 4-7 DUTs each and much more custom handling - such as hot-unplugging the device from the VM, resetting and firmware updating the device, and hot-plugging it back as part of the test running in that VM - as testing PCIe devices themselves exercise many more issues that you don't see with standardized hardware. I have done this for about a decade, so I've been through a few iterations and tech stacks. One can find things that work, but it's not in any way or form guaranteed to work.