4 ms·

> One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of car

> One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards and still get all the bandwidth out of a 16-lane CPU. PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them. Even that use case has been eroded a bit with PCIe splitting options, where an x16 lane can be split into x8/x8 or even x4/x4/x4/x4 depending on the motherboard. An x4 PCIe 4.0 lane is as fast as an old x16 PCIe 2.0 lane, so that's a significant amount of bandwidth still. PCIe 5.0 takes this even further, where an x16 PCIe 5.0 slot has over 60GB/sec of bandwidth, which is a lot even when divided into x4 or x2 channels. AMD especially made platforms with a lot of PCIe lanes relatively affordable. An AMD workstation motherboard and CPU with a lot of PCIe lanes might not actually be that much more expensive than a hypothetical consumer CPU with a PCIe switch. The other issue is that with a PCIe switch is that you're still limited to upstream bandwidth, so if you need x16 bandwidth running to multiple cards at the same time, a PCIe switch doesn't actually help you.

bilegeek 4y ago
It's still a problem if you want to run previous gen hardware. For instance, if I have a Gen4 motherboard and pass through a second GPU to a VM - so the second GPU at Gen3 x8 plus the host GPU at Gen4 x8 - I'm only using a quarter instead of half the throughput the hardware actually supports, because Gen3 is half the speed of Gen4 per lane. Or even think about Gen5, doubling Gen4 speeds: With a PCIe switch, I could get essentially 3/4 performance out of a Gen5 host GPU, and full performance out of a Gen3 second, instead of 1/2 and 1/4 respectively.
- Dalewyn 4y ago
  Do GPUs even saturate PCIe3 x16 or x8 connections, let alone PCIe4 or 5, that we need to be worried about bottlenecks? The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.
  - KennyBlanken 4y ago
    For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.
    - bilegeek 4y ago
      Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates. Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind. [1]https://looking-glass.io/ https://looking-glass.io/
      - wtallis 4y ago
        To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.
  - theblazehen 4y ago
    I've had significant frame rate drops with PCIe3 x8 compared to PCIe3 x16 (Easy 20 - 40% drop, depending on the game)
  - bick_nyers 4y ago
    If I remember correctly, running a 3080 10GB in PCIe 4.0 yields >5% performance average in gaming benchmarks compared to running it in PCIe 3.0. You could probably argue that you should just settle and not take the 5%, but when you are buying in at such a price point, you are likely concerned with 5% differences. There are also higher end SKUs such as the 3090Ti where that performance gap is likely larger.
- bick_nyers 4y ago
  This is precisely why I am not going to buy into this year's Ryzen motherboards, how am I supposed to attach all of my PCIe 2.0 LSI HBA as well as my 2 GPUs, and NVMe? My options are buying 2 machines (one for storage, one for compute), sacrificing speed somewhere, or buying into either Threadripper Pro OEM or EPYC ($$$). The bandwidth is there by an order of magnitude, but not the lanes to actually use it.
fomine3 4y ago
It's pain that even PCIe splitting isn't adopted well on recent Z690/X670 mobo. So they just provide us a PCIe 5.0 x16 slot that is unnecessary great but practically nothing better than 3.0 x16 for gaming (and most workload) now . I'd like to have PCIe 4.0 x8/x8 rather than that.
- c_o_n_v_e_x 4y ago
  PCIe bifurcation is a function of what your CPU and/or PCH can do. There are hard limits in the silicon as to how you can divvy up your lanes.
  - fomine3 4y ago
    Splitting CPU lane to x8/x8 is supported at least. Just mobo manufacturers don't want to implement it as a dedicated slot. It seems that Alder Lake no longer support x8/x4/x4 sadly.
    - wtallis 4y ago
      > It seems that Alder Lake no longer support x8/x4/x4 sadly. I would be very surprised if the silicon doesn't still support 8+4+4 operation. I suspect the lack of available motherboards with this config comes down to the expense of getting a PCIe gen5 signal to reach the third slot. It may require both the extra mux and a redriver or retimer.
      - fomine3 4y ago
        I found this from quick Googling https://forum-en.msi.com/index.php?threads/does-pro-z690-a-and-mag-z690-tomahawks-pcie-5-0-x16-slots-support-pcie-bifurcation.375664/ https://forum-en.msi.com/index.php?threads/does-pro-z690-a-a... Perhaps why they dropped the support is because the reason you wrote? Still I wonder is it possible to have Gen5 x8 and Gen3 x4/x4.
    - eigen 4y ago
      > It seems that Alder Lake no longer support x8/x4/x4 sadly. Sort of. support is "Up to 1x16+4, 2x8+4" [1] so could be used as x8/x4/x4, the remaining 4 lanes are unused in the x8, as there are 20 PCIe lanes on the CPU. [1] https://ark.intel.com/content/www/us/en/ark/products/228441/intel-core-i912900hx-processor-30m-cache-up-to-5-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/228441/...
c_o_n_v_e_x 4y ago
>PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them. PCIe switches still have a place in storage servers. I can build a backplane with a PCIe switch so I can get a ton of ports/slots for HDDs or SSDs and not have to buy a much more expensive CPU with the IO to handle everything. The trade off is that you are over subscribed and all disks will never reach full bandwidth simultaneously.

bilegeek 4y ago

It's still a problem if you want to run previous gen hardware. For instance, if I have a Gen4 motherboard and pass through a second GPU to a VM - so the second GPU at Gen3 x8 plus the host GPU at Gen4 x8 - I'm only using a quarter instead of half the throughput the hardware actually supports, because Gen3 is half the speed of Gen4 per lane. Or even think about Gen5, doubling Gen4 speeds: With a PCIe switch, I could get essentially 3/4 performance out of a Gen5 host GPU, and full performance out of a Gen3 second, instead of 1/2 and 1/4 respectively.

Dalewyn 4y ago
Do GPUs even saturate PCIe3 x16 or x8 connections, let alone PCIe4 or 5, that we need to be worried about bottlenecks? The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.
- KennyBlanken 4y ago
  For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.
  - bilegeek 4y ago
    Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates. Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind. [1]https://looking-glass.io/ https://looking-glass.io/
    - wtallis 4y ago
      To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.
- theblazehen 4y ago
  I've had significant frame rate drops with PCIe3 x8 compared to PCIe3 x16 (Easy 20 - 40% drop, depending on the game)
- bick_nyers 4y ago
  If I remember correctly, running a 3080 10GB in PCIe 4.0 yields >5% performance average in gaming benchmarks compared to running it in PCIe 3.0. You could probably argue that you should just settle and not take the 5%, but when you are buying in at such a price point, you are likely concerned with 5% differences. There are also higher end SKUs such as the 3090Ti where that performance gap is likely larger.
bick_nyers 4y ago
This is precisely why I am not going to buy into this year's Ryzen motherboards, how am I supposed to attach all of my PCIe 2.0 LSI HBA as well as my 2 GPUs, and NVMe? My options are buying 2 machines (one for storage, one for compute), sacrificing speed somewhere, or buying into either Threadripper Pro OEM or EPYC ($$$). The bandwidth is there by an order of magnitude, but not the lanes to actually use it.

fomine3 4y ago

It's pain that even PCIe splitting isn't adopted well on recent Z690/X670 mobo. So they just provide us a PCIe 5.0 x16 slot that is unnecessary great but practically nothing better than 3.0 x16 for gaming (and most workload) now . I'd like to have PCIe 4.0 x8/x8 rather than that.

c_o_n_v_e_x 4y ago
PCIe bifurcation is a function of what your CPU and/or PCH can do. There are hard limits in the silicon as to how you can divvy up your lanes.
- fomine3 4y ago
  Splitting CPU lane to x8/x8 is supported at least. Just mobo manufacturers don't want to implement it as a dedicated slot. It seems that Alder Lake no longer support x8/x4/x4 sadly.
  - wtallis 4y ago
    > It seems that Alder Lake no longer support x8/x4/x4 sadly. I would be very surprised if the silicon doesn't still support 8+4+4 operation. I suspect the lack of available motherboards with this config comes down to the expense of getting a PCIe gen5 signal to reach the third slot. It may require both the extra mux and a redriver or retimer.
    - fomine3 4y ago
      I found this from quick Googling https://forum-en.msi.com/index.php?threads/does-pro-z690-a-and-mag-z690-tomahawks-pcie-5-0-x16-slots-support-pcie-bifurcation.375664/ https://forum-en.msi.com/index.php?threads/does-pro-z690-a-a... Perhaps why they dropped the support is because the reason you wrote? Still I wonder is it possible to have Gen5 x8 and Gen3 x4/x4.
  - eigen 4y ago
    > It seems that Alder Lake no longer support x8/x4/x4 sadly. Sort of. support is "Up to 1x16+4, 2x8+4" [1] so could be used as x8/x4/x4, the remaining 4 lanes are unused in the x8, as there are 20 PCIe lanes on the CPU. [1] https://ark.intel.com/content/www/us/en/ark/products/228441/intel-core-i912900hx-processor-30m-cache-up-to-5-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/228441/...

c_o_n_v_e_x 4y ago

>PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them. PCIe switches still have a place in storage servers. I can build a backplane with a PCIe switch so I can get a ton of ports/slots for HDDs or SSDs and not have to buy a much more expensive CPU with the IO to handle everything. The trade off is that you are over subscribed and all disks will never reach full bandwidth simultaneously.