4 ms·

I worked on a PCIE Gen3 retimer. The problem was you were trying to fool both ends into thinking you weren't there and was kind of violating the spec. PCIE Ge

I worked on a PCIE Gen3 retimer. The problem was you were trying to fool both ends into thinking you weren't there and was kind of violating the spec. PCIE Gen4 has explicitly defined the terms allowing you to make a chip that is compliant with the spec to retime the signal.

segfaultbuserr 6y ago
> fool both ends into thinking you weren't there and was kind of violating the spec. Long-range USB (e.g. for conference rooms) has the same problem. The maximum one-way propagation delay is a hard restriction in the standard. You have to somehow cheat the host into thinking the device is busy before the peripheral at the far side is able to respond. To this day, I believe no standard solution exists, it seems everyone's reinventing the wheel in their FPGAs. Although it's only a niche application, standardization is unlikely [0]. BTW, FireWire did have a perfect solution to this problem because good networking was one of its design goals. [0] And for relatively shorter runs it's not usually a problem - cascading USB hubs is good enough, it's how most active extension cables work - as one-port hubs.
- marcan_42 6y ago
  USB sucks. Bigtime. Way, way more than PCIe. PCIe is great. PCIe will run over wet string* . USB... USB is a world of pain. This is the same problem any kind of USB virtualization/network tunnelling has. Also, anyone trying to build a USB2 to USB3 transaction translator (which the spec, unforgivably, omitted). AFAIK there is one USB2 to USB3 TT chip in the market and it's unobtainium. It works if you lift things back to the URB level to some extent, and "virtualize" the lower protocol layers, but there are corner cases once you introduce a frame of latency or more, which you have to. The host polls for data, then you're polling the device for data. You get back some data, but the next frame the host has stopped polling for data. Now you have some data you have to drop on the floor. Not good. You could choose to delay a successful ACK until you get it from the other side (and thus pretend like the first time the data was sent it was corrupted, even though it was forwarded successfully), but now you've massively decimated your maximum throughput. The same problem happens in the other direction too. Thankfully, in practice these race conditions often hit software anyway (i.e. if you cancel a submitted URB then if data came back it was already acked and it'll be dropped on the floor), so you can get away with it since drivers should be designed not to screw this up. But it's still easier at the software level (e.g. virtualizing) because you actually know what the host is trying to do. I wrote the QEMU virtual xHCI code and the original prototype, which ignored the QEMU USB subsystem and passed through straight to the host, worked very well (better than anything VMware/virtualbox were doing) because xHCI is high level enough. But at the wire level, you don't even know when the host has cancelled a transaction other than just... because it chooses not to send packets in a given frame... and what if it just decided not to poll that frame but it will poll later? Now you need timeout heuristics... it's horrible. * seriously, I've tunnelled PCIe over 115200 baud RS232. You can do all kinds of horrible things to PCIe and it will still work.
  - segfaultbuserr 6y ago
    > USB sucks. PCIe is great. > This is the same problem any kind of USB virtualization/network tunnelling has. +1. I've been using QubesOS on my workstation for a while, and I found the problem associated with USB virtualization/tunnelling also has a major impact on QubesOS's usability (USBility?). Long story short, QubesOS's security model depends on the ability to isolate individual peripherals at the hypervisor level. For PCI-E it's easy, but for USB it's a headache. It has various security, compatibility and performance problems. If one has ever encountered such a problem, the only workaround is assigning an entire EHCI or xHCI USB host controller at the PCI-E level to a VM, and this is only feasible on machines with multiple USB controllers. > I wrote the QEMU virtual xHCI code and the original prototype, which ignored the QEMU USB subsystem and passed through straight to the host, worked very well I also got the idea to build something similar, but in hardware - put 4 individual PCI-E USB host controllers behind a PCI-E switch, so that each USB port can be seen as an individual PCI-E device. Now the USB problem in Qubes is solved forever...
    - pedrocr 6y ago
      Isn't the PCIe card you're looking for this: https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Express-USB-3-Card-4-Dedicated-Channels-4-Port~PEXUSB3S44V https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Ex... Seems to have 4 USB ports each served by it's own controller with dedicated PCIe lanes.
      - segfaultbuserr 6y ago
        Yes, this is exactly what I was describing, and it can be the solution. However, I was thinking of a downscaled USB 2.0 version - only has a fraction of its cost, and it doesn't eat all my PCI-E lanes (20 Gbps vs 1.92 Gbps, the latter one only needs a PCI-E x1 Gen 2).
  - Dylan16807 6y ago
    > seriously, I've tunnelled PCIe over 115200 baud RS232. You can do all kinds of horrible things to PCIe and it will still work. Ah, yeah, I remember this presentation: https://fail0verflow.com/media/33c3-slides/#/11 https://fail0verflow.com/media/33c3-slides/#/11

segfaultbuserr 6y ago

> fool both ends into thinking you weren't there and was kind of violating the spec. Long-range USB (e.g. for conference rooms) has the same problem. The maximum one-way propagation delay is a hard restriction in the standard. You have to somehow cheat the host into thinking the device is busy before the peripheral at the far side is able to respond. To this day, I believe no standard solution exists, it seems everyone's reinventing the wheel in their FPGAs. Although it's only a niche application, standardization is unlikely [0]. BTW, FireWire did have a perfect solution to this problem because good networking was one of its design goals. [0] And for relatively shorter runs it's not usually a problem - cascading USB hubs is good enough, it's how most active extension cables work - as one-port hubs.

marcan_42 6y ago
USB sucks. Bigtime. Way, way more than PCIe. PCIe is great. PCIe will run over wet string* . USB... USB is a world of pain. This is the same problem any kind of USB virtualization/network tunnelling has. Also, anyone trying to build a USB2 to USB3 transaction translator (which the spec, unforgivably, omitted). AFAIK there is one USB2 to USB3 TT chip in the market and it's unobtainium. It works if you lift things back to the URB level to some extent, and "virtualize" the lower protocol layers, but there are corner cases once you introduce a frame of latency or more, which you have to. The host polls for data, then you're polling the device for data. You get back some data, but the next frame the host has stopped polling for data. Now you have some data you have to drop on the floor. Not good. You could choose to delay a successful ACK until you get it from the other side (and thus pretend like the first time the data was sent it was corrupted, even though it was forwarded successfully), but now you've massively decimated your maximum throughput. The same problem happens in the other direction too. Thankfully, in practice these race conditions often hit software anyway (i.e. if you cancel a submitted URB then if data came back it was already acked and it'll be dropped on the floor), so you can get away with it since drivers should be designed not to screw this up. But it's still easier at the software level (e.g. virtualizing) because you actually know what the host is trying to do. I wrote the QEMU virtual xHCI code and the original prototype, which ignored the QEMU USB subsystem and passed through straight to the host, worked very well (better than anything VMware/virtualbox were doing) because xHCI is high level enough. But at the wire level, you don't even know when the host has cancelled a transaction other than just... because it chooses not to send packets in a given frame... and what if it just decided not to poll that frame but it will poll later? Now you need timeout heuristics... it's horrible. * seriously, I've tunnelled PCIe over 115200 baud RS232. You can do all kinds of horrible things to PCIe and it will still work.
- segfaultbuserr 6y ago
  > USB sucks. PCIe is great. > This is the same problem any kind of USB virtualization/network tunnelling has. +1. I've been using QubesOS on my workstation for a while, and I found the problem associated with USB virtualization/tunnelling also has a major impact on QubesOS's usability (USBility?). Long story short, QubesOS's security model depends on the ability to isolate individual peripherals at the hypervisor level. For PCI-E it's easy, but for USB it's a headache. It has various security, compatibility and performance problems. If one has ever encountered such a problem, the only workaround is assigning an entire EHCI or xHCI USB host controller at the PCI-E level to a VM, and this is only feasible on machines with multiple USB controllers. > I wrote the QEMU virtual xHCI code and the original prototype, which ignored the QEMU USB subsystem and passed through straight to the host, worked very well I also got the idea to build something similar, but in hardware - put 4 individual PCI-E USB host controllers behind a PCI-E switch, so that each USB port can be seen as an individual PCI-E device. Now the USB problem in Qubes is solved forever...
  - pedrocr 6y ago
    Isn't the PCIe card you're looking for this: https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Express-USB-3-Card-4-Dedicated-Channels-4-Port~PEXUSB3S44V https://www.startech.com/Cards-Adapters/USB-3.0/Cards/PCI-Ex... Seems to have 4 USB ports each served by it's own controller with dedicated PCIe lanes.
    - segfaultbuserr 6y ago
      Yes, this is exactly what I was describing, and it can be the solution. However, I was thinking of a downscaled USB 2.0 version - only has a fraction of its cost, and it doesn't eat all my PCI-E lanes (20 Gbps vs 1.92 Gbps, the latter one only needs a PCI-E x1 Gen 2).
- Dylan16807 6y ago
  > seriously, I've tunnelled PCIe over 115200 baud RS232. You can do all kinds of horrible things to PCIe and it will still work. Ah, yeah, I remember this presentation: https://fail0verflow.com/media/33c3-slides/#/11 https://fail0verflow.com/media/33c3-slides/#/11