3 ms·

I've never given much thought to how much PCIe lanes are available to my hardware. However, with the advent of NVME drives I realise that there are only so many

I've never given much thought to how much PCIe lanes are available to my hardware. However, with the advent of NVME drives I realise that there are only so many drives one can install.

arghwhat 5y ago
PCIe switches are a thing, and devices don't need all their lanes if you're okay with less max bandwidth. It ends up being the same as any other interface: there's a limit to how many devices can be connected before the limit for non-blocking simultaneous bandwidth is reached, but that merely means that bandwidth can be a bottleneck if you go above that.
- dogma1138 5y ago
  Below switches which on the high end can have their own root complex, the more common thing is lane bifurcation and multiplexing. This holds especially true for consumer motherboards and PCIe breakout boards.
  - arghwhat 5y ago
    Speaking from experience: bifurcation is very uncommon and support is absolutely terrible, with most motherboards lacking the feature or having it be broken. Even when it is supported it's usually a pain... Multiplexers just lets you switch between connected devices - it won't let you use both simultaneously. It's hot-swapping without having to physically change devices.
    - dogma1138 5y ago
      Bifurcation is much better supported these days due to the prevalence of mining. And yes multiplexing is basically “time sharing” and the PLX chips operate on the physical layer only. PCIe switches have an internal bus, buffers and actual decode the packets to know where to send them too, they can also mediate between different versions of PCIe connected to the same switch so connecting a PCIe 2.0 device to a switch would not impact other 3.0/4.0 devices whilst a multiplexer would always operate at the lowest “speed”. But PLX chips were pretty much the only thing you can get on consumer motherboards at least back when multi GPU setups were common and SLI/XFire motherboards we’re a thing (that said PLX did help quite a bit more with XFire setups than SLI due to the lack of an external cross bridge between GPUs in later XFire revisions). The chipset on your motherboard can also support its own PCIe lanes however at least on Intel chipsets it’s not a classical PCIe switch its closer to the silicon in the CPU that runs the root complex.
kkielhofner 5y ago
This is a crucial consideration when choosing hardware (especially Intel vs AMD) [0]. Of note is the AMD Threadripper family supporting up to 56 PCIe lanes directly to the CPU. On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. Massive amounts of I/O such as this are instrumental for high and extremely dense I/O applications whether it be for storage, network, GPUs, etc (or combinations of all, of course). I know it's been a consideration for companies such as Netflix, Cloudflare, and Nvidia. [0] https://www.cgdirector.com/guide-to-pcie-lanes/ https://www.cgdirector.com/guide-to-pcie-lanes/ [1] https://www.amd.com/en/processors/epyc-7002-series https://www.amd.com/en/processors/epyc-7002-series
- cesarb 5y ago
  > On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. However, AMD Epyc uses half of the 128 lanes on each socket to talk to the other socket, so on two-socket systems each socket has only 64 PCIe lanes available, for a total of 128 lanes, the same as with a single socket. On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. The article at https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-128-160-pcie-gen4-lanes-and-a-bonus/ https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-... has a great explanation of all that.
  - kkielhofner 5y ago
    Oh wow! I didn't know that. I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? I still remember building a two socket Xeon workstation some years ago and being puzzled that video wasn't initializing. Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that?

I've never given much thought to how much PCIe lanes are available to my hardware. However, with the advent of NVME drives I realise that there are only so many drives one can install.

arghwhat 5y ago

PCIe switches are a thing, and devices don't need all their lanes if you're okay with less max bandwidth. It ends up being the same as any other interface: there's a limit to how many devices can be connected before the limit for non-blocking simultaneous bandwidth is reached, but that merely means that bandwidth can be a bottleneck if you go above that.

dogma1138 5y ago
Below switches which on the high end can have their own root complex, the more common thing is lane bifurcation and multiplexing. This holds especially true for consumer motherboards and PCIe breakout boards.
- arghwhat 5y ago
  Speaking from experience: bifurcation is very uncommon and support is absolutely terrible, with most motherboards lacking the feature or having it be broken. Even when it is supported it's usually a pain... Multiplexers just lets you switch between connected devices - it won't let you use both simultaneously. It's hot-swapping without having to physically change devices.
  - dogma1138 5y ago
    Bifurcation is much better supported these days due to the prevalence of mining. And yes multiplexing is basically “time sharing” and the PLX chips operate on the physical layer only. PCIe switches have an internal bus, buffers and actual decode the packets to know where to send them too, they can also mediate between different versions of PCIe connected to the same switch so connecting a PCIe 2.0 device to a switch would not impact other 3.0/4.0 devices whilst a multiplexer would always operate at the lowest “speed”. But PLX chips were pretty much the only thing you can get on consumer motherboards at least back when multi GPU setups were common and SLI/XFire motherboards we’re a thing (that said PLX did help quite a bit more with XFire setups than SLI due to the lack of an external cross bridge between GPUs in later XFire revisions). The chipset on your motherboard can also support its own PCIe lanes however at least on Intel chipsets it’s not a classical PCIe switch its closer to the silicon in the CPU that runs the root complex.

kkielhofner 5y ago

This is a crucial consideration when choosing hardware (especially Intel vs AMD) [0]. Of note is the AMD Threadripper family supporting up to 56 PCIe lanes directly to the CPU. On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. Massive amounts of I/O such as this are instrumental for high and extremely dense I/O applications whether it be for storage, network, GPUs, etc (or combinations of all, of course). I know it's been a consideration for companies such as Netflix, Cloudflare, and Nvidia. [0] https://www.cgdirector.com/guide-to-pcie-lanes/ https://www.cgdirector.com/guide-to-pcie-lanes/ [1] https://www.amd.com/en/processors/epyc-7002-series https://www.amd.com/en/processors/epyc-7002-series

cesarb 5y ago
> On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. However, AMD Epyc uses half of the 128 lanes on each socket to talk to the other socket, so on two-socket systems each socket has only 64 PCIe lanes available, for a total of 128 lanes, the same as with a single socket. On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. The article at https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-128-160-pcie-gen4-lanes-and-a-bonus/ https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-... has a great explanation of all that.
- kkielhofner 5y ago
  Oh wow! I didn't know that. I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? I still remember building a two socket Xeon workstation some years ago and being puzzled that video wasn't initializing. Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that?