3 ms·
it's more than 6x the bandwidth of practical network speeds today.
by shaklee3 4y ago
it's more than 6x the bandwidth of practical network speeds today.
- touisteur 4y agoI would have compared with PCIe bandwidth, which seem to drag behind actual network speeds these days. In the PCIe 3 era, Mellanox/NVIDIA sold (still sell) 200g (2x100 mostly) boards using 2 PCIe3 16x ports, linked by a specific cable (the whole montage is called SocketDirect? or sometimes PCIe 32x?). When PCIe 4 started getting standardized you could already saturate the bus with one 200G NIC. Now ConnectX 7 is out, should be able to saturate PCIe 5 but most constructors haven't got their motherboards out. SFP112 is out and you can already saturate PCIe 6 with 2 QSFP112 (or equivalent 2x4x100G). Still need CPUs with support or GPUs with PCIe5 or 6 support, which I guess are coming or you can stream from one NIC to several GPUs. It might be more interesting for nvidia in the mid to long run to not wait for PCIe standards to settle and to allow streaming to/from their NICs to their GPUs (or CPUs, see Grace Hopper white paper) using nvlink, never touching the PCIe bus...
- shaklee3 4y agothe problem is PCIe speeds are increasing faster than network. pcie5 can hit 500Gbps in theory, but the network is only 400Gbps.