3 ms·

Please forgive my lack of familiarity with PCI express hardware, but I have a question: All those PCI express devices are all sharing the single v2 lane? Is th

Please forgive my lack of familiarity with PCI express hardware, but I have a question: All those PCI express devices are all sharing the single v2 lane? Is the PCI express bus a network?

geerlingguy 5y ago
Indeed, PCI Express can be thought of in many ways like Ethernet; you can have switches just like Ethernet, though in almost all cases, PCIe is a local area bus (local to one server/machine), whereas Ethernet is still the king of networks.
NovemberWhiskey 5y ago
That's probably the way to think about it. PCIe devices communicate point-to-point over dual-simplex serial connections. If you have something like a Raspberry Pi CM4, which only has a single PCIe lane, then there's going to be a PCIe switch.
- bloopernova 5y ago
  Interesting, thank you!
  - Teknoman117 5y ago
    The other neat thing is that peer-to-peer PCIe transactions do not have to go through the root complex (the Pi in this case). Say you have a GPU and an NVMe drive or video capture card plugged into a PCIe switch, which is then plugged into the Pi. If your software supports having these devices directly communicate, they can do so as fast as their PCIe links support doing, even though there’s only the tiny link to the Pi. Peer-to-peer transfers are only limited by the smallest link on the path between two devices on the tree, rather than the smallest link in the tree. A real world example is some of the big GPU compute servers. Some of the 4U chassis can house 20 or more cards. Many of them have all of the GPUs (320 or more PCIe lanes) plugged into an array of PCIe switches which may constrain them to 16 or 32 lanes from the CPU. The GPUs can all talk to each other a full speed, but only a few can go full speed to the CPU at a time.
    - salawat 5y ago
      Don't you mean the CPU can only full-duplex with a few GPU's at a time? And are you sure that GPU's are actually talking with each other? Forgive me if I'm woefully out of date/having a massive brainfart, but last I heard, GPU's got CPU-less DMA a few years ago allowing access to system memory, but programming/setting up the pipelines/cleanup/repriming/output consolidation was still solely a CPU-centric work stream. So like, yeah, your GPU does it's thing and dumps the result in a buffer in it's memory space, and might even be smart enough to shlup a buffer back to system RAM to get the result forwarded somewhere else when the CPU gets around to telling the storage controller to do it... But did I completely sleep through something absolutely amazing? Are GPU's passing messages/shared memory space aware now? Like, "Oh, my jobs done, shlup this buffer off to pcie device->compute_card_2 for the next stage of processing." Because if they're that smart now, holy crap, I have reading to do.
      - Teknoman117 5y ago
        Yes, GPUs are able to directly pass messages and bulk data to each other these days. It's completely supported in CUDA, OpenCL, Vulkan, DirectX 12, etc. DirectX calls it "Explicit Multi GPU". It's been around since GPUs transitioned to PCIe, it's just that both AMD and NVIDIA just locked it to the "professional" GPUs, or only used it under the hood of some other feature. AMD GPUs have used PCIe peer to peer copies for CrossFireX since the GCN Radeon GPUs (R9 290X and friends), that's why they don't need an external bridge. There's DirectGMA (AMD) and GPUDirect (NVIDIA) which allows PCIe drivers for other devices to directly copy to and from GPU VRAM. Other than driver support, nothing prevents PCs from having GPUs communicate directly with storage devices. The GPU would need to expose some of it's VRAM over PCIe (using a BAR window). The OS and the GPU driver would need to create some NVMe submission and completion queues either in system or VRAM that the GPU can be considered to own. You'd then issue NVMe operations where the backing pages point into the VRAM. The resizable bar support of the current Radeon and GeForce GPUs is a major step in this direction, although again, resizable bars were part of the original PCIe specification and has been available on server hardware since ~2008.
        5y ago
dragontamer 5y ago
All I/O is a network these days. * USB is a network. * PCIe is a network. * SATA is a network. * Infinity Fabric / Hypertransport (AMD socket-to-socket communications) is a network. * UltraPath Interconnect (Intel socket-to-socket communications) is a network. ----------- People think their internet connection is Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer1 (Wires), but that's wrong. Its really Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer 3 (PCIe) -> Layer 2 (PCIe frames) -> Layer 1 (PCIe wires) -> reinterpret PCIe on the Ethernet-controller -> Layer1 (Ethernet wires). There's a bunch of "networks" in-between our networks that we use. That's why OSI model exists, to understand each different network and each of the different rules behind them all. When a good network is abstracted away, you can ignore all the details. EDIT: It might not even be Layer2(Ethernet) -> Layer3 (PCIe)... on a NUMA-system it could be Layer2(Ethernet) -> Layer2 (NUMA / MESI messages to CPU#2) -> Layer2 (Southbridge communications) -> Layer3(PCIe)...
- agumonkey 5y ago
  network all the way down -- Alan Kay maybe btw, slight sidestep HDMI CEC is also a bus/network, and even SCART was too.
colejohnson66 5y ago
One of the goals of PCIe when they were designing it was to avoid the “bus” architecture classic PCI had as that forced every device on the bus to run at the speed of the slowest device. PCIe, OTOH, takes a “star” network topology (more a "tree" though) with point to point connections. Thanks to the use of switches, each point can run at the maximum speed they support (limited by the switches speed). Basically, it’s like USB (as @dragontamer said). Just as you can turn a single USB port into 7 more downstream ports through a USB switch, you can do the same with a PCIe switch.

Please forgive my lack of familiarity with PCI express hardware, but I have a question: All those PCI express devices are all sharing the single v2 lane? Is the PCI express bus a network?

geerlingguy 5y ago

Indeed, PCI Express can be thought of in many ways like Ethernet; you can have switches just like Ethernet, though in almost all cases, PCIe is a local area bus (local to one server/machine), whereas Ethernet is still the king of networks.

NovemberWhiskey 5y ago

That's probably the way to think about it. PCIe devices communicate point-to-point over dual-simplex serial connections. If you have something like a Raspberry Pi CM4, which only has a single PCIe lane, then there's going to be a PCIe switch.

bloopernova 5y ago
Interesting, thank you!
- Teknoman117 5y ago
  The other neat thing is that peer-to-peer PCIe transactions do not have to go through the root complex (the Pi in this case). Say you have a GPU and an NVMe drive or video capture card plugged into a PCIe switch, which is then plugged into the Pi. If your software supports having these devices directly communicate, they can do so as fast as their PCIe links support doing, even though there’s only the tiny link to the Pi. Peer-to-peer transfers are only limited by the smallest link on the path between two devices on the tree, rather than the smallest link in the tree. A real world example is some of the big GPU compute servers. Some of the 4U chassis can house 20 or more cards. Many of them have all of the GPUs (320 or more PCIe lanes) plugged into an array of PCIe switches which may constrain them to 16 or 32 lanes from the CPU. The GPUs can all talk to each other a full speed, but only a few can go full speed to the CPU at a time.
  - salawat 5y ago
    Don't you mean the CPU can only full-duplex with a few GPU's at a time? And are you sure that GPU's are actually talking with each other? Forgive me if I'm woefully out of date/having a massive brainfart, but last I heard, GPU's got CPU-less DMA a few years ago allowing access to system memory, but programming/setting up the pipelines/cleanup/repriming/output consolidation was still solely a CPU-centric work stream. So like, yeah, your GPU does it's thing and dumps the result in a buffer in it's memory space, and might even be smart enough to shlup a buffer back to system RAM to get the result forwarded somewhere else when the CPU gets around to telling the storage controller to do it... But did I completely sleep through something absolutely amazing? Are GPU's passing messages/shared memory space aware now? Like, "Oh, my jobs done, shlup this buffer off to pcie device->compute_card_2 for the next stage of processing." Because if they're that smart now, holy crap, I have reading to do.
    - Teknoman117 5y ago
      Yes, GPUs are able to directly pass messages and bulk data to each other these days. It's completely supported in CUDA, OpenCL, Vulkan, DirectX 12, etc. DirectX calls it "Explicit Multi GPU". It's been around since GPUs transitioned to PCIe, it's just that both AMD and NVIDIA just locked it to the "professional" GPUs, or only used it under the hood of some other feature. AMD GPUs have used PCIe peer to peer copies for CrossFireX since the GCN Radeon GPUs (R9 290X and friends), that's why they don't need an external bridge. There's DirectGMA (AMD) and GPUDirect (NVIDIA) which allows PCIe drivers for other devices to directly copy to and from GPU VRAM. Other than driver support, nothing prevents PCs from having GPUs communicate directly with storage devices. The GPU would need to expose some of it's VRAM over PCIe (using a BAR window). The OS and the GPU driver would need to create some NVMe submission and completion queues either in system or VRAM that the GPU can be considered to own. You'd then issue NVMe operations where the backing pages point into the VRAM. The resizable bar support of the current Radeon and GeForce GPUs is a major step in this direction, although again, resizable bars were part of the original PCIe specification and has been available on server hardware since ~2008.
      - 5y ago

dragontamer 5y ago

All I/O is a network these days. * USB is a network. * PCIe is a network. * SATA is a network. * Infinity Fabric / Hypertransport (AMD socket-to-socket communications) is a network. * UltraPath Interconnect (Intel socket-to-socket communications) is a network. ----------- People think their internet connection is Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer1 (Wires), but that's wrong. Its really Layer4 (TCP) -> Layer3 (IP) -> Layer2 (Ethernet) -> Layer 3 (PCIe) -> Layer 2 (PCIe frames) -> Layer 1 (PCIe wires) -> reinterpret PCIe on the Ethernet-controller -> Layer1 (Ethernet wires). There's a bunch of "networks" in-between our networks that we use. That's why OSI model exists, to understand each different network and each of the different rules behind them all. When a good network is abstracted away, you can ignore all the details. EDIT: It might not even be Layer2(Ethernet) -> Layer3 (PCIe)... on a NUMA-system it could be Layer2(Ethernet) -> Layer2 (NUMA / MESI messages to CPU#2) -> Layer2 (Southbridge communications) -> Layer3(PCIe)...

agumonkey 5y ago
network all the way down -- Alan Kay maybe btw, slight sidestep HDMI CEC is also a bus/network, and even SCART was too.

colejohnson66 5y ago

One of the goals of PCIe when they were designing it was to avoid the “bus” architecture classic PCI had as that forced every device on the bus to run at the speed of the slowest device. PCIe, OTOH, takes a “star” network topology (more a "tree" though) with point to point connections. Thanks to the use of switches, each point can run at the maximum speed they support (limited by the switches speed). Basically, it’s like USB (as @dragontamer said). Just as you can turn a single USB port into 7 more downstream ports through a USB switch, you can do the same with a PCIe switch.