3 ms·
Consumer cards are PCIe 4.0 x16. H100 PCIe version is PCIe 5.0 https://en.wikipedia.org/wiki/Hopper_(microarchitecture) https://en.wikipedia.org/wiki/Hopper_(m
by alecco 2y ago
Consumer cards are PCIe 4.0 x16. H100 PCIe version is PCIe 5.0 https://en.wikipedia.org/wiki/Hopper_(microarchitecture) https://en.wikipedia.org/wiki/Hopper_(microarchitecture). And it's been out 2 years already.
- throwway120385 2y agoSo everyone is supposed to do all of their testing on H100's?
- alecco 2y ago4090 (2022) PCIe 4.0 x16 is quite decent. The major limit is memory, not bandwidth. And 3090 (2020) is also PCIe 4.0 x16, and used cards are a bargain. You can hook them up with Nvlink. Nvidia is withholding new releases but the current hardware has more legs with new matrix implementations. Like FlashAttention doing some significant improvement every 6 months. Nvidia could make consumer chips with combined CPU-GPU. I guess they are too busy making money with the big cloud. Maybe somebody will pick up. Apple is already doing something like it even on laptops.
- _zoltan_ 2y agoget a GH100 on lambda and behold you have 900GB/s between CPU memory and GPU, and forget PCIe.
- saagarjha 2y agoWhere are you seeing 900 GB/s?
- KeplerBoy 2y agoThe 900 GB/s are a figure often cited for Hopper based SXM boards, it's the aggregate of 18 NVLink connections. So it's more of a many-to-many GPU-to-GPU bandwidth figure. https://www.datacenterknowledge.com/data-center-hardware/nvidia-s-h100-what-it-is-what-it-does-and-why-it-matters https://www.datacenterknowledge.com/data-center-hardware/nvi...
- shaklee3 2y agothis is bidirectional bw and it's gh200, not gh100
- imtringued 2y agoThat doesn't add up, because you're now exceeding the memory bandwidth of the memory controller. I.e. it would be faster to do everything, including the CPU only algorithms, in far away VRAM.
- gpderetta 2y agolatency might be higher and latency usually dominates CPU side algorithms.
- jsheard 2y agoTIL, I missed Hopper already having it. I assume the RTX 5000 series will bring it to consumers.
- yvdriess 2y agoFor the consumer GPUs, PCIe 4.0 x16 has plenty of BW headroom. The full sized x16 is more for stability reasons. Some vendors even put a couple of M.2 slots on PCI4/5 GPU board to recuperate the unused PCIe lanes.
- deleted 2y ago[deleted]
- KeplerBoy 2y agoThat's mixing apples with oranges. Some low-end or middle range GPUs really only use 8 lanes, because that's how the chip is designed. Fewer lanes -> less silicon area for the pcie logic needed -> cheaper. The chips which use 16 lanes take advantage of it and can saturate the link.
- touisteur 2y agoWell let me tell you about the pretty high-end and expensive L40 that shipped with PCIe-4.0 to my utter dismay and disgust. Only the H100 had 5.0 although I could already saturate 4.0 (and 5.0) with Mellanox NICs and GPU/StorageDirect. Waiting for the next one to maybe get 5.0.