5 ms·

Here I am still on PCI-E 3.0...

Arnavion 2y ago
Most hardware (NVMe drives, GPUs, etc) doesn't run at more than 4.0 speeds anyway. The primary advantage of 5.0 and higher is that it'll allow that hardware to use fewer CPU lanes, eg what requires 4.0 x4 could use 6.0 x1.
- accrual 2y ago
  It kinda worked that way in the past too. IIRC, newer AGP graphics cards would be keyed for 4x or 8x slots but would barely use more bandwidth than the original 2x slots provided.
  - lightedman 2y ago
    We didn't really bottleneck AGP8X until nVidia dropped the 200 series GeForce GPU. Then PCI-E was pretty much a requirement.
- steelbrain 2y ago
  > eg what requires 4.0 x4 could use 6.0 x1 FWIW, this is only true for newer hardware. ie if you plugged in a pcie gen3x16 device into a pcie gen4x8 slot, although the bandwidth provided is in the same ballpark, the device will only run at pcie gen3x8. So we'll need until the devices upgrade themselves to gen4 in this scenario to make use of higher bandwidth.
  - Arnavion 2y ago
    I'm not saying the device itself would negotiate a higher PCIe version. I'm saying that the 4.0 x4 M.2 NVMe slot on your mobo would map to only one 6.0 CPU lane.
    - seritools 2y ago
      Huh, I was pretty sure that you need an extra chip in-between. Otherwise, the 4 CPU lanes will just drop to 4.0 level.
      - Arnavion 2y ago
        Yes you do.
        Dylan16807 2y ago
        You'd need a bunch of those chips to make good use of a bunch of 6.0 lanes on the CPU, and at that point you're paying so much for the converters that I'm skeptical it would almost ever be worth it. I expect consumer machines to keep doing some conversion and expansion in the chipset, but nowhere else. I expect servers to directly attach almost everything and drop down to smaller lane counts for large numbers of devices. It's worth noting that when Kioxia first put out PCIe 5.0 EDSFF drives, they were marketing them as being optimized for 2 lanes at the higher speed.
        deleted 2y ago
        [deleted]
  - KeplerBoy 2y ago
    It could still help connecting the chipset. Not all pci-e lanes on your motherboard are created equal: Some are directly attached to the CPU others are connected to the chipset, which in turn is connected to the CPU. It's possible to convert a single 5.0 x16 connection coming from the CPU to 2 4.0 x16 connections.
- oblio 2y ago
  I can we can stack them? If we had 8x NVMe drives bundled I assume we'd need PCI 7.
daemonologist 2y ago
It felt like we were on 3 for a long time, and then all of a sudden got 4 through 6 (and soon 7) in quick succession. I'd be curious to know what motivated that - maybe GPGPU taking off?
- latchkey 2y ago
  AI/GPU communication is definitely driving it forward now. It is a speed race for how quickly you can move data around.
  - starspangled 2y ago
    Really? I hadn't heard of GPU or GPGPU pushing bandwidth recently. Networking certainly does. 400GbE cards exceed PCIe 4.0 x16 bandwidth, 800 is here, and 1.6 apparently in the works. Disk too though, just because a single disk (or even network phy) may not max out a PCI slot does not mean you want to dedicate more lanes than necessary to them because you likely want a bunch of them.
    - latchkey 2y ago
      We are at PCIe5 in the Dell XE9680. We add in 8x400G cards and they talk directly to the Network/ 8xGPUs (via rocev2). 800G ethernet is here at the switches (Dell Z9864F-ON is beautiful... 128 ports of 400G), but not yet at the server/NIC level, that comes with PCIe6. We are also limited to 16 chassis/128 GPUs in a single cluster right now. NVMe is getting faster all the time, but is pretty standard now. We put 122TB into each server, so that enables local caching of data, if needed. All of this is designed for the highest speed available today that we can get on the various bus where data is transferred.
      - oblio 2y ago
        I wonder if any of this trickles down into cloud providers reducing costs again. After all if we have zounds of fast storage, surely slower storage becomes cheaper?
Sohcahtoa82 2y ago
Same. I upgraded to a 4 TB NVMe drive that theoretically reads/writes at up to 7.7 GB/s, but I only get 3.5 GB/s because my CPU is still an i9-9900K running PCI-E 3.0. Planning on upgrading once the next Intel generation drops.

Most hardware (NVMe drives, GPUs, etc) doesn't run at more than 4.0 speeds anyway. The primary advantage of 5.0 and higher is that it'll allow that hardware to use fewer CPU lanes, eg what requires 4.0 x4 could use 6.0 x1.

accrual 2y ago
It kinda worked that way in the past too. IIRC, newer AGP graphics cards would be keyed for 4x or 8x slots but would barely use more bandwidth than the original 2x slots provided.
- lightedman 2y ago
  We didn't really bottleneck AGP8X until nVidia dropped the 200 series GeForce GPU. Then PCI-E was pretty much a requirement.
steelbrain 2y ago
> eg what requires 4.0 x4 could use 6.0 x1 FWIW, this is only true for newer hardware. ie if you plugged in a pcie gen3x16 device into a pcie gen4x8 slot, although the bandwidth provided is in the same ballpark, the device will only run at pcie gen3x8. So we'll need until the devices upgrade themselves to gen4 in this scenario to make use of higher bandwidth.
- Arnavion 2y ago
  I'm not saying the device itself would negotiate a higher PCIe version. I'm saying that the 4.0 x4 M.2 NVMe slot on your mobo would map to only one 6.0 CPU lane.
  - seritools 2y ago
    Huh, I was pretty sure that you need an extra chip in-between. Otherwise, the 4 CPU lanes will just drop to 4.0 level.
    - Arnavion 2y ago
      Yes you do.
      - Dylan16807 2y ago
        You'd need a bunch of those chips to make good use of a bunch of 6.0 lanes on the CPU, and at that point you're paying so much for the converters that I'm skeptical it would almost ever be worth it. I expect consumer machines to keep doing some conversion and expansion in the chipset, but nowhere else. I expect servers to directly attach almost everything and drop down to smaller lane counts for large numbers of devices. It's worth noting that when Kioxia first put out PCIe 5.0 EDSFF drives, they were marketing them as being optimized for 2 lanes at the higher speed.
        deleted 2y ago
        [deleted]
- KeplerBoy 2y ago
  It could still help connecting the chipset. Not all pci-e lanes on your motherboard are created equal: Some are directly attached to the CPU others are connected to the chipset, which in turn is connected to the CPU. It's possible to convert a single 5.0 x16 connection coming from the CPU to 2 4.0 x16 connections.
oblio 2y ago
I can we can stack them? If we had 8x NVMe drives bundled I assume we'd need PCI 7.

daemonologist 2y ago

It felt like we were on 3 for a long time, and then all of a sudden got 4 through 6 (and soon 7) in quick succession. I'd be curious to know what motivated that - maybe GPGPU taking off?

latchkey 2y ago
AI/GPU communication is definitely driving it forward now. It is a speed race for how quickly you can move data around.
- starspangled 2y ago
  Really? I hadn't heard of GPU or GPGPU pushing bandwidth recently. Networking certainly does. 400GbE cards exceed PCIe 4.0 x16 bandwidth, 800 is here, and 1.6 apparently in the works. Disk too though, just because a single disk (or even network phy) may not max out a PCI slot does not mean you want to dedicate more lanes than necessary to them because you likely want a bunch of them.
  - latchkey 2y ago
    We are at PCIe5 in the Dell XE9680. We add in 8x400G cards and they talk directly to the Network/ 8xGPUs (via rocev2). 800G ethernet is here at the switches (Dell Z9864F-ON is beautiful... 128 ports of 400G), but not yet at the server/NIC level, that comes with PCIe6. We are also limited to 16 chassis/128 GPUs in a single cluster right now. NVMe is getting faster all the time, but is pretty standard now. We put 122TB into each server, so that enables local caching of data, if needed. All of this is designed for the highest speed available today that we can get on the various bus where data is transferred.
    - oblio 2y ago
      I wonder if any of this trickles down into cloud providers reducing costs again. After all if we have zounds of fast storage, surely slower storage becomes cheaper?

Sohcahtoa82 2y ago

Same. I upgraded to a 4 TB NVMe drive that theoretically reads/writes at up to 7.7 GB/s, but I only get 3.5 GB/s because my CPU is still an i9-9900K running PCI-E 3.0. Planning on upgrading once the next Intel generation drops.