4 ms·

The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs: 20 lanes from CPU and a few more from the SouthBridge is not enough when the G

The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs: 20 lanes from CPU and a few more from the SouthBridge is not enough when the GPU is usually linked on a 16 lane connection. The easy way out is to reduce GPU lanes to 8, that leaves plenty of bandwidth for nVME SSDs and maybe for 10 or 25 Gbps NICs (it's about time). For servers it is a different story, but the recent fast move from PCIe ver 3 to ver 5 improved the situation 4x, doubling again is nice, but it does not seem that much of a deal. Maybe moving NICs from the usual 8 lane to a lot less (8 lanes of ver 3 means 2 lanes of ver 5 or a single lane of ver 6) will also make some difference.

alberth 5y ago
Isn't this only an artificial problem Intel created to segment the market, a problem that AMD doesn't have.
- csdvrx 5y ago
  Yes they do have that problem, even the upcoming AM5 will only have 28 lanes given the annoucements: https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-120-w-more-pcie-lanes-cpu-image/ https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-12...
- wmf 5y ago
  AMD has the same or slightly more lanes than Intel.
  - the8472 5y ago
    EPYCs have 128 PCIe Gen4, recent Xeons have 64 PCIe Gen4. And Intel introduced Gen4 later than AMD.
    - zamadatix 5y ago
      > The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs... For servers it is a different story EPYC/Threadripper is awesome but also explicitly not the segment being discussed. But since it was mentioned it's 64 PCIe lanes per socket on Intel (up to 8 sockets/512 lanes) instead of 128 lanes for single socket/up to 160 lanes for dual socket on Epyc. Gonna cost you a damn arm and leg and you better know your NUMA though. In the consumer segment AMD has 16 PCIe lanes for the graphics and up to 4 PCIe lanes for NVMe drives, the chipset link is and additional x4 PCIe instead of a different interconnect. This comes to 20 PCIe lanes for user devices and 4 lanes worth of PCIe bandwidth off the chipset. https://i.imgur.com/8Aug02l.png https://i.imgur.com/8Aug02l.png Intel offers something similar, 16 PCIe lanes for graphics and 4 PCIe lanes for NVMe. They opt for a proprietary DMI connection to the chipset which is equivalent to 8 lanes of PCIe 3.0 bandwidth or 4 lanes of PCIe 3.0 bandwith. https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-6.fit_lim.size_768x.png https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-... Each offers splitting the x16 for the GPU into 2 x8 as well as oversubscribing the downstream PCIe bandwidth from the chipset, devices through the chipset on each obviously hit different latency penalties than CPU direct lanes as well. In the end the offerings are both pretty much identical in the consumer space.
- tjoff 5y ago
  It's an AMD problem as well. It's absolute nightmare trying to research for a computer today. What ports can you use in what circumstances. Which slots go the CPU directly and which go to a chipset. Which lanes are disabled if you use nvme-slot 2. Which slot has which generation etc. A proper nightmare. And while we are at it, dedicating pci-lanes to nvme-slots must be one of the most boneheaded decisions in modern computers. Just use a pci-card with up to four nvme-slots on it instead.
  - FridgeSeal 5y ago
    Maybe it’s because I bought a “gaming” motherboard, but the manual was pretty (for my understanding at least) as to what configuration of m.2 drives and PCIe lanes would run at what version, what went to cpu and what went to chipset.
    - deckard1 5y ago
      The issue is cross comparison, not reading from the manual of a single board. In the past you could look at the physical motherboard layout and could compare it with another motherboard. Today, it's merely a suggestion. You take the physical layout with a huge grain of salt because it's nothing but lies. Which means going to the website of every single potential motherboard you wish to purchase, downloading their PDF, and digging through enough caveat fine print documentation to make your eyes bleed.
  - nijave 5y ago
    A card with more than 1 NVMe slot requirements PCIe bifurcation which not everything supports
    - tjoff 5y ago
      That's only because it's a niche feature. If that was the official way to do it then it would be supported by everything.
- jeffbee 5y ago
  It's not just segmentation. Laptop buyers are not going to pay for 64 lanes. A regular Intel SKU of the 12th generation has 28 PCIe 4.0/5.0 lanes. A Xeon has 64, does not have 5.0, and costs way more, partly because it has 4189 pins on the bottom, which is insane.
  - Dylan16807 5y ago
    https://www.intel.com/content/www/us/en/processors/core/desktop-6th-gen-core-family-datasheet-vol-1.html https://www.intel.com/content/www/us/en/processors/core/desk... Based on page 131 of this, it looks like a PCIe lane only uses up six pins. I'd worry about the die space and the power use but not the pin use.
addaon 5y ago
But doubling the bandwidth per lane allows one to use half as many lanes to a GPU and maintain bandwidth. As you mention, it allows an eight lane GPU to be a viable option. And better yet, due to how PCIe handles variable number of lanes between host and device, different users with the same CPU, GPU, and even motherboard can choose to run the GPU at eight lane with a couple of four-lane SSDs, or at sixteen lane for even more bandwidth if they don't need that bandwidth elsewhere.
- johncolanduoni 5y ago
  PCIe bandwidth increases have outstripped increases in GPU bandwidth use by games for a while now. Anything more than 8x is overkill unless you’re doing GPGPU work: https://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-performance-impact-on-gpus https://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-pe...
- AdrianB1 5y ago
  8 lane GPU is viable for a long time (benchmarks on PCIe 8x versus 16x shows a 5% perf difference), but it did not change the physical layout the motherboard manufacturers use; you cannot use the existing lanes any way you want, on some motherboards you cannot even split it the way you want between physical connectors and video card manufacturers continue to push 16x everywhere.
tjoff 5y ago
With the bandwidth you do get more flexibility though, and with pci express bifurcation you can get the bandwidth equivalent of 4 pcie4 16x from a single pcie6 16x. And that's great since the vast majority don't need that much bandwidth anyway. Today you typically have a whole slew of devices sharing 4x to the cpu. More bandwidth would open up for more usb and perhaps cheaper onboard 10gig Ethernet etc.
csdvrx 5y ago
Totally, and sTRX4 has a limited set of board available. I was hoping AM4 would provide that many lanes on easy to buy motherboards, but it's a meager 28, so not even enough for 2x x16
- yxhuvud 5y ago
  At least AM5, which is coming soon, seems to improve the situation.
  - csdvrx 5y ago
    With a meagre extra 4 lanes

alberth 5y ago

Isn't this only an artificial problem Intel created to segment the market, a problem that AMD doesn't have.

csdvrx 5y ago
Yes they do have that problem, even the upcoming AM5 will only have 28 lanes given the annoucements: https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-120-w-more-pcie-lanes-cpu-image/ https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-12...
wmf 5y ago
AMD has the same or slightly more lanes than Intel.
- the8472 5y ago
  EPYCs have 128 PCIe Gen4, recent Xeons have 64 PCIe Gen4. And Intel introduced Gen4 later than AMD.
  - zamadatix 5y ago
    > The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs... For servers it is a different story EPYC/Threadripper is awesome but also explicitly not the segment being discussed. But since it was mentioned it's 64 PCIe lanes per socket on Intel (up to 8 sockets/512 lanes) instead of 128 lanes for single socket/up to 160 lanes for dual socket on Epyc. Gonna cost you a damn arm and leg and you better know your NUMA though. In the consumer segment AMD has 16 PCIe lanes for the graphics and up to 4 PCIe lanes for NVMe drives, the chipset link is and additional x4 PCIe instead of a different interconnect. This comes to 20 PCIe lanes for user devices and 4 lanes worth of PCIe bandwidth off the chipset. https://i.imgur.com/8Aug02l.png https://i.imgur.com/8Aug02l.png Intel offers something similar, 16 PCIe lanes for graphics and 4 PCIe lanes for NVMe. They opt for a proprietary DMI connection to the chipset which is equivalent to 8 lanes of PCIe 3.0 bandwidth or 4 lanes of PCIe 3.0 bandwith. https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-6.fit_lim.size_768x.png https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-... Each offers splitting the x16 for the GPU into 2 x8 as well as oversubscribing the downstream PCIe bandwidth from the chipset, devices through the chipset on each obviously hit different latency penalties than CPU direct lanes as well. In the end the offerings are both pretty much identical in the consumer space.
tjoff 5y ago
It's an AMD problem as well. It's absolute nightmare trying to research for a computer today. What ports can you use in what circumstances. Which slots go the CPU directly and which go to a chipset. Which lanes are disabled if you use nvme-slot 2. Which slot has which generation etc. A proper nightmare. And while we are at it, dedicating pci-lanes to nvme-slots must be one of the most boneheaded decisions in modern computers. Just use a pci-card with up to four nvme-slots on it instead.
- FridgeSeal 5y ago
  Maybe it’s because I bought a “gaming” motherboard, but the manual was pretty (for my understanding at least) as to what configuration of m.2 drives and PCIe lanes would run at what version, what went to cpu and what went to chipset.
  - deckard1 5y ago
    The issue is cross comparison, not reading from the manual of a single board. In the past you could look at the physical motherboard layout and could compare it with another motherboard. Today, it's merely a suggestion. You take the physical layout with a huge grain of salt because it's nothing but lies. Which means going to the website of every single potential motherboard you wish to purchase, downloading their PDF, and digging through enough caveat fine print documentation to make your eyes bleed.
- nijave 5y ago
  A card with more than 1 NVMe slot requirements PCIe bifurcation which not everything supports
  - tjoff 5y ago
    That's only because it's a niche feature. If that was the official way to do it then it would be supported by everything.
jeffbee 5y ago
It's not just segmentation. Laptop buyers are not going to pay for 64 lanes. A regular Intel SKU of the 12th generation has 28 PCIe 4.0/5.0 lanes. A Xeon has 64, does not have 5.0, and costs way more, partly because it has 4189 pins on the bottom, which is insane.
- Dylan16807 5y ago
  https://www.intel.com/content/www/us/en/processors/core/desktop-6th-gen-core-family-datasheet-vol-1.html https://www.intel.com/content/www/us/en/processors/core/desk... Based on page 131 of this, it looks like a PCIe lane only uses up six pins. I'd worry about the die space and the power use but not the pin use.

addaon 5y ago

But doubling the bandwidth per lane allows one to use half as many lanes to a GPU and maintain bandwidth. As you mention, it allows an eight lane GPU to be a viable option. And better yet, due to how PCIe handles variable number of lanes between host and device, different users with the same CPU, GPU, and even motherboard can choose to run the GPU at eight lane with a couple of four-lane SSDs, or at sixteen lane for even more bandwidth if they don't need that bandwidth elsewhere.

johncolanduoni 5y ago
PCIe bandwidth increases have outstripped increases in GPU bandwidth use by games for a while now. Anything more than 8x is overkill unless you’re doing GPGPU work: https://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-performance-impact-on-gpus https://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-pe...
AdrianB1 5y ago
8 lane GPU is viable for a long time (benchmarks on PCIe 8x versus 16x shows a 5% perf difference), but it did not change the physical layout the motherboard manufacturers use; you cannot use the existing lanes any way you want, on some motherboards you cannot even split it the way you want between physical connectors and video card manufacturers continue to push 16x everywhere.

tjoff 5y ago

With the bandwidth you do get more flexibility though, and with pci express bifurcation you can get the bandwidth equivalent of 4 pcie4 16x from a single pcie6 16x. And that's great since the vast majority don't need that much bandwidth anyway. Today you typically have a whole slew of devices sharing 4x to the cpu. More bandwidth would open up for more usb and perhaps cheaper onboard 10gig Ethernet etc.

csdvrx 5y ago

Totally, and sTRX4 has a limited set of board available. I was hoping AM4 would provide that many lanes on easy to buy motherboards, but it's a meager 28, so not even enough for 2x x16

yxhuvud 5y ago
At least AM5, which is coming soon, seems to improve the situation.
- csdvrx 5y ago
  With a meagre extra 4 lanes