7 ms·

I love the PCIe standard is 3 generations ahead of what is actually released. Gen5 is the live version, but the team behind it is so well organized that they h

I love the PCIe standard is 3 generations ahead of what is actually released. Gen5 is the live version, but the team behind it is so well organized that they have a roadmap of 3 additional versions now. Love it.

ThatMedicIsASpy 1y ago
Gen6 is in use look at Nvidia ConnectX-8
- drewg123 1y ago
  What hosts support Gen6? AFAIK, Gen5 is the most recent standard that's actually deployed. Eg, what can you plug a CX8 into that will link up at Gen6?
  - triknomeister 1y ago
    Custom Nvidia network cards I guess.
  - my123 1y ago
    Blackwell DC (B200/B300)
    - _zoltan_ 1y ago
      while the B200 chip itself could do PCIe6 as it was planned for GB200, there is no system around it with Gen6. the official DGX B200 is just PCIe5. GB300 is indeed Gen6.
  - _zoltan_ 1y ago
    into a Gen6 Bianca board (baseboard for GB300). that's it today. how you use those today is twofold: - Gen5 x32 via two x16 slots (this is how most people use them) - use actually the CX8 as your PCIe switch directly to your GPUs: https://www.servethehome.com/this-is-the-nvidia-mgx-pcie-switch-board-with-connectx-8-for-8x-pcie-gpu-servers/ https://www.servethehome.com/this-is-the-nvidia-mgx-pcie-swi...
  - ksec 1y ago
    Nvidia only for now. Precisely because they got fed up with Intel and AMD ( And Amphere as well ) for extremely slow PCIe adoption. I believe only next Gen Intel and AMD Zen6 will get PCIe 6.0. I am hoping Nvidia officially move into Server CPU market not only for their CPU but for wider Web Hosting as well. More competition for Server Hardware.
tails4e 1y ago
It takes a long time to get form standard to silicon, so I bet there are design teams working on pcie7 right now, which won't see products for 2 or more years
Seattle3503 1y ago
Is there an advantage of getting so far ahead of implementations? It seems like it would be more difficult to incorporate lessons.
- kvemkon 1y ago
  When AMD introduces a new Desktop CPU series IIRC they claim the next generation design is (almost) finished (including layout?) and they start with the next-next-gen design. And I'm also asking the same question. But more than a half a year before the CPU becomes available to the public it is already being tested by partners (mainboard manufacturers and ?).
Phelinofist 1y ago
So we can skip 6 and 7 and go directly to 8, right?
zamadatix 1y ago
"3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today, PCIe 7.x was finalized last month, and this is an announcement work on PCIe 8.0 has started for release in 3 years. I.e. it has only been one month of being one generation behind the latest PCIe revision. It'll be interesting if consumer devices bother trying to stay with the latest at all anymore. It's already extremely difficult to justify the cost of implementing PCIe 5.0 when it makes almost no difference for consumer use cases. The best consumer use case so far is enthusiasts who want really fast NVMe SSDs in x4 lanes, but 5.0 already gives >10 GB/s for a single drive, even with the limited lane count. It makes very little difference for x16 GPUs, even with the 5090. Things always creep up over time, but the rate at which the consumer space creeps is just so vastly different from what the DC space has been seeing that it seems unreasonable to expect the two to be lockstep anymore.
- _zoltan_ 1y ago
  either I'm misinformed in which case is like to see proof, or you are, but GB200 is not Gen6, just Gen5 and this will only be rectified with GB300 which, while exists in the wild, is not in the millions. so indeed the parent commenter would be correct that everything is Gen5 right now.
  - zamadatix 1y ago
    I'm going off of https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/?utm_source=chatgpt.com#:~:text=The%20GB200%20has%20cold%20plates%20and%20connections%20for%20liquid%20cooling%2C%20PCIe%20gen%206%20support https://developer.nvidia.com/blog/nvidia-gb200-nvl72-deliver... but I have not actually deployed a GB200 system yet myself. Consumer Blackwell is definitely 5.0 though.
    - _zoltan_ 1y ago
      that article is from last March. the plan was indeed for the GB200 board to support gen6 but it didn't happen and GB300 will rectify it. at least that's my best educated guess, looking at supermicro's public spec sheet and that it's shipping with CX7 which is Gen5 and not with CX8. B200 supports Gen6 there is just nothing that would let it run at Gen6.
jsolson 1y ago
This actually makes sense from a spec perspective if you want to give enough to allow hardware to catch up with the specs and to support true interop. Contrast this with the wild west that is "Ethernet" where it's extremely common for speeds to track well ahead of specs and where interop is, at best, "exciting."

ThatMedicIsASpy 1y ago

Gen6 is in use look at Nvidia ConnectX-8

drewg123 1y ago
What hosts support Gen6? AFAIK, Gen5 is the most recent standard that's actually deployed. Eg, what can you plug a CX8 into that will link up at Gen6?
- triknomeister 1y ago
  Custom Nvidia network cards I guess.
- my123 1y ago
  Blackwell DC (B200/B300)
  - _zoltan_ 1y ago
    while the B200 chip itself could do PCIe6 as it was planned for GB200, there is no system around it with Gen6. the official DGX B200 is just PCIe5. GB300 is indeed Gen6.
- _zoltan_ 1y ago
  into a Gen6 Bianca board (baseboard for GB300). that's it today. how you use those today is twofold: - Gen5 x32 via two x16 slots (this is how most people use them) - use actually the CX8 as your PCIe switch directly to your GPUs: https://www.servethehome.com/this-is-the-nvidia-mgx-pcie-switch-board-with-connectx-8-for-8x-pcie-gpu-servers/ https://www.servethehome.com/this-is-the-nvidia-mgx-pcie-swi...
- ksec 1y ago
  Nvidia only for now. Precisely because they got fed up with Intel and AMD ( And Amphere as well ) for extremely slow PCIe adoption. I believe only next Gen Intel and AMD Zen6 will get PCIe 6.0. I am hoping Nvidia officially move into Server CPU market not only for their CPU but for wider Web Hosting as well. More competition for Server Hardware.

tails4e 1y ago

It takes a long time to get form standard to silicon, so I bet there are design teams working on pcie7 right now, which won't see products for 2 or more years

Seattle3503 1y ago

Is there an advantage of getting so far ahead of implementations? It seems like it would be more difficult to incorporate lessons.

kvemkon 1y ago
When AMD introduces a new Desktop CPU series IIRC they claim the next generation design is (almost) finished (including layout?) and they start with the next-next-gen design. And I'm also asking the same question. But more than a half a year before the CPU becomes available to the public it is already being tested by partners (mainboard manufacturers and ?).

Phelinofist 1y ago

So we can skip 6 and 7 and go directly to 8, right?

zamadatix 1y ago

"3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today, PCIe 7.x was finalized last month, and this is an announcement work on PCIe 8.0 has started for release in 3 years. I.e. it has only been one month of being one generation behind the latest PCIe revision. It'll be interesting if consumer devices bother trying to stay with the latest at all anymore. It's already extremely difficult to justify the cost of implementing PCIe 5.0 when it makes almost no difference for consumer use cases. The best consumer use case so far is enthusiasts who want really fast NVMe SSDs in x4 lanes, but 5.0 already gives >10 GB/s for a single drive, even with the limited lane count. It makes very little difference for x16 GPUs, even with the 5090. Things always creep up over time, but the rate at which the consumer space creeps is just so vastly different from what the DC space has been seeing that it seems unreasonable to expect the two to be lockstep anymore.

_zoltan_ 1y ago
either I'm misinformed in which case is like to see proof, or you are, but GB200 is not Gen6, just Gen5 and this will only be rectified with GB300 which, while exists in the wild, is not in the millions. so indeed the parent commenter would be correct that everything is Gen5 right now.
- zamadatix 1y ago
  I'm going off of https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/?utm_source=chatgpt.com#:~:text=The%20GB200%20has%20cold%20plates%20and%20connections%20for%20liquid%20cooling%2C%20PCIe%20gen%206%20support https://developer.nvidia.com/blog/nvidia-gb200-nvl72-deliver... but I have not actually deployed a GB200 system yet myself. Consumer Blackwell is definitely 5.0 though.
  - _zoltan_ 1y ago
    that article is from last March. the plan was indeed for the GB200 board to support gen6 but it didn't happen and GB300 will rectify it. at least that's my best educated guess, looking at supermicro's public spec sheet and that it's shipping with CX7 which is Gen5 and not with CX8. B200 supports Gen6 there is just nothing that would let it run at Gen6.

jsolson 1y ago

This actually makes sense from a spec perspective if you want to give enough to allow hardware to catch up with the specs and to support true interop. Contrast this with the wild west that is "Ethernet" where it's extremely common for speeds to track well ahead of specs and where interop is, at best, "exciting."