4 ms·

"3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today, PCIe 7.x was finalized last month, and this is an announcement

"3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today, PCIe 7.x was finalized last month, and this is an announcement work on PCIe 8.0 has started for release in 3 years. I.e. it has only been one month of being one generation behind the latest PCIe revision. It'll be interesting if consumer devices bother trying to stay with the latest at all anymore. It's already extremely difficult to justify the cost of implementing PCIe 5.0 when it makes almost no difference for consumer use cases. The best consumer use case so far is enthusiasts who want really fast NVMe SSDs in x4 lanes, but 5.0 already gives >10 GB/s for a single drive, even with the limited lane count. It makes very little difference for x16 GPUs, even with the 5090. Things always creep up over time, but the rate at which the consumer space creeps is just so vastly different from what the DC space has been seeing that it seems unreasonable to expect the two to be lockstep anymore.

_zoltan_ 1y ago
either I'm misinformed in which case is like to see proof, or you are, but GB200 is not Gen6, just Gen5 and this will only be rectified with GB300 which, while exists in the wild, is not in the millions. so indeed the parent commenter would be correct that everything is Gen5 right now.
- zamadatix 1y ago
  I'm going off of https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/?utm_source=chatgpt.com#:~:text=The%20GB200%20has%20cold%20plates%20and%20connections%20for%20liquid%20cooling%2C%20PCIe%20gen%206%20support https://developer.nvidia.com/blog/nvidia-gb200-nvl72-deliver... but I have not actually deployed a GB200 system yet myself. Consumer Blackwell is definitely 5.0 though.
  - _zoltan_ 1y ago
    that article is from last March. the plan was indeed for the GB200 board to support gen6 but it didn't happen and GB300 will rectify it. at least that's my best educated guess, looking at supermicro's public spec sheet and that it's shipping with CX7 which is Gen5 and not with CX8. B200 supports Gen6 there is just nothing that would let it run at Gen6.
    - zamadatix 1y ago
      At least the GB200s OpenAI purchased have CX-8s, but it's very possible (perhaps even very likely) most of the millions of GB200s deployed PCIe 5.0 using CX-7s given the SuperMicro spec sheet. Edit: found this SemiAnalysis post saying as much https://x.com/SemiAnalysis_/status/1947768988467138645/photo/1 https://x.com/SemiAnalysis_/status/1947768988467138645/photo...
      - _zoltan_ 1y ago
        thanks! very interesting that they got a custom SKU.
  - jsolson 1y ago
    Datacenter Blackwell is Gen6, which is critical when pairing it with CX8 (2x400G) as otherwise you'd be stranding NIC BW.
    - _zoltan_ 1y ago
      please link me a B200 (not a B300) system with CX8 that uses gen6 x16 instead of gen5 x32. to the best of my knowledge this does not exist, but I'd be happy to stand corrected. (the official NVIDIA DGX B200 is gen5).
Melatonic 1y ago
CPUs themselves only have so many PCI-E lanes though right? Wouldnt it make sense (even for consumers) to have peripherals using less lanes (but more speed per lane) for a multi GPU system or something that uses a lot of drives?
- zamadatix 1y ago
  More lanes = more cost Faster lanes = more cost More faster lanes = lots more cost The chipset also strikes some of the balance for consumers though. It has a narrow high speed connection to the CPU but enables many lower speed devices to share that bandwidth. That way you can have your spare NVMe drive, SATA controller, wired and wireless NICs, sound hardware, most of your USB ports, your capture card, and some other random things connected over a single x4 to x8 sized channel. This leaves the high cost lanes for just the devices that actually use them (GPU and primary, possibly secondary, storage drive). I've got one consumer type Motherboard with 14 NVMe drives connected, for example, just not at full native speed directly to the CPU. You're just SoL if you want to connect a bunch of really high bandwidth devices simultaneously (100 Gbps+ NICs, multiple GPUs at full connection speed, a dozen NVMe drives at native speed, or similar) because then you'll be paying for a workstation/server class platform which did make the "more faster lanes" tradeoff (plus some market segment gouging, of course).
  - vladvasiliu 1y ago
    One issue is that, at least on cheaper mobos, these don't work as a "total bandwidth budget" situation. And, especially with newer generation PCIe, it can be a bit frustrating. Many mobos will operate the available slots such that the total number of active lanes is split between them. But if you use older-generation cards, you'll only get a fraction of the available bandwidth because you're only using a fraction of their lanes, although the physical lanes are physically present. What I'm thinking about is something like, say, a pair of Gen3 NVMe drives that are good enough for mass storage (running in RAID-1 for good measure) and some cheap used 10 Gb NIC, which will probably be 8x Gen2, all running on a Gen4+ capable mobo. And, while for a general-purpose setup I can live with splitting available BW between the NIC and the GPU (I most likely don't care about my download going super fast while I game), the downloads will generally go to the storage, so they must be fast at the same time.
michaelt 1y ago
> "3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today True. And yet, if you buy an RTX 5090 today, costing $2400 and released in January this year, it's PCIe 5.0 x16
- zamadatix 1y ago
  Does the second half not already talk to this? Adding on to it further, GN found a few % performance difference on a 5090 going from PCIe 3.0 to 5.0 https://gamersnexus.net/gpus/nvidia-rtx-5090-pcie-50-vs-40-vs-30-x16-scaling-benchmarks https://gamersnexus.net/gpus/nvidia-rtx-5090-pcie-50-vs-40-v... meanwhile many DC use cases find PCIe 5.0 unusably slow, driving alternative (non-PCIe) interconnects.

_zoltan_ 1y ago

either I'm misinformed in which case is like to see proof, or you are, but GB200 is not Gen6, just Gen5 and this will only be rectified with GB300 which, while exists in the wild, is not in the millions. so indeed the parent commenter would be correct that everything is Gen5 right now.

zamadatix 1y ago
I'm going off of https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/?utm_source=chatgpt.com#:~:text=The%20GB200%20has%20cold%20plates%20and%20connections%20for%20liquid%20cooling%2C%20PCIe%20gen%206%20support https://developer.nvidia.com/blog/nvidia-gb200-nvl72-deliver... but I have not actually deployed a GB200 system yet myself. Consumer Blackwell is definitely 5.0 though.
- _zoltan_ 1y ago
  that article is from last March. the plan was indeed for the GB200 board to support gen6 but it didn't happen and GB300 will rectify it. at least that's my best educated guess, looking at supermicro's public spec sheet and that it's shipping with CX7 which is Gen5 and not with CX8. B200 supports Gen6 there is just nothing that would let it run at Gen6.
  - zamadatix 1y ago
    At least the GB200s OpenAI purchased have CX-8s, but it's very possible (perhaps even very likely) most of the millions of GB200s deployed PCIe 5.0 using CX-7s given the SuperMicro spec sheet. Edit: found this SemiAnalysis post saying as much https://x.com/SemiAnalysis_/status/1947768988467138645/photo/1 https://x.com/SemiAnalysis_/status/1947768988467138645/photo...
    - _zoltan_ 1y ago
      thanks! very interesting that they got a custom SKU.
- jsolson 1y ago
  Datacenter Blackwell is Gen6, which is critical when pairing it with CX8 (2x400G) as otherwise you'd be stranding NIC BW.
  - _zoltan_ 1y ago
    please link me a B200 (not a B300) system with CX8 that uses gen6 x16 instead of gen5 x32. to the best of my knowledge this does not exist, but I'd be happy to stand corrected. (the official NVIDIA DGX B200 is gen5).

Melatonic 1y ago

CPUs themselves only have so many PCI-E lanes though right? Wouldnt it make sense (even for consumers) to have peripherals using less lanes (but more speed per lane) for a multi GPU system or something that uses a lot of drives?

zamadatix 1y ago
More lanes = more cost Faster lanes = more cost More faster lanes = lots more cost The chipset also strikes some of the balance for consumers though. It has a narrow high speed connection to the CPU but enables many lower speed devices to share that bandwidth. That way you can have your spare NVMe drive, SATA controller, wired and wireless NICs, sound hardware, most of your USB ports, your capture card, and some other random things connected over a single x4 to x8 sized channel. This leaves the high cost lanes for just the devices that actually use them (GPU and primary, possibly secondary, storage drive). I've got one consumer type Motherboard with 14 NVMe drives connected, for example, just not at full native speed directly to the CPU. You're just SoL if you want to connect a bunch of really high bandwidth devices simultaneously (100 Gbps+ NICs, multiple GPUs at full connection speed, a dozen NVMe drives at native speed, or similar) because then you'll be paying for a workstation/server class platform which did make the "more faster lanes" tradeoff (plus some market segment gouging, of course).
- vladvasiliu 1y ago
  One issue is that, at least on cheaper mobos, these don't work as a "total bandwidth budget" situation. And, especially with newer generation PCIe, it can be a bit frustrating. Many mobos will operate the available slots such that the total number of active lanes is split between them. But if you use older-generation cards, you'll only get a fraction of the available bandwidth because you're only using a fraction of their lanes, although the physical lanes are physically present. What I'm thinking about is something like, say, a pair of Gen3 NVMe drives that are good enough for mass storage (running in RAID-1 for good measure) and some cheap used 10 Gb NIC, which will probably be 8x Gen2, all running on a Gen4+ capable mobo. And, while for a general-purpose setup I can live with splitting available BW between the NIC and the GPU (I most likely don't care about my download going super fast while I game), the downloads will generally go to the storage, so they must be fast at the same time.

michaelt 1y ago

> "3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today True. And yet, if you buy an RTX 5090 today, costing $2400 and released in January this year, it's PCIe 5.0 x16

zamadatix 1y ago
Does the second half not already talk to this? Adding on to it further, GN found a few % performance difference on a 5090 going from PCIe 3.0 to 5.0 https://gamersnexus.net/gpus/nvidia-rtx-5090-pcie-50-vs-40-vs-30-x16-scaling-benchmarks https://gamersnexus.net/gpus/nvidia-rtx-5090-pcie-50-vs-40-v... meanwhile many DC use cases find PCIe 5.0 unusably slow, driving alternative (non-PCIe) interconnects.