3 ms·

Can anyone elaborate on the pros and cons of nvlink in comparison to e.g. rdma over infiniband/roce, or pci-e?

• woodson 4y ago
  According to the article, it has more than three times the bandwidth of PCI-E Gen5 per lane.
• deleted 4y ago
  [deleted]
• wmf 4y ago
  I think NVLink is a memory interconnect more like NUMAlink, UPI, or Infinity Fabric. AFAIK GPUs can transparently access each other's memory over NVLink (probably at cache line granularity) while RDMA protocols require explicitly copying memory (usually in larger chunks) before accessing it.
• shaklee3 4y ago
  it's more than 6x the bandwidth of practical network speeds today.
  • touisteur 4y ago
    I would have compared with PCIe bandwidth, which seem to drag behind actual network speeds these days. In the PCIe 3 era, Mellanox/NVIDIA sold (still sell) 200g (2x100 mostly) boards using 2 PCIe3 16x ports, linked by a specific cable (the whole montage is called SocketDirect? or sometimes PCIe 32x?). When PCIe 4 started getting standardized you could already saturate the bus with one 200G NIC. Now ConnectX 7 is out, should be able to saturate PCIe 5 but most constructors haven't got their motherboards out. SFP112 is out and you can already saturate PCIe 6 with 2 QSFP112 (or equivalent 2x4x100G). Still need CPUs with support or GPUs with PCIe5 or 6 support, which I guess are coming or you can stream from one NIC to several GPUs. It might be more interesting for nvidia in the mid to long run to not wait for PCIe standards to settle and to allow streaming to/from their NICs to their GPUs (or CPUs, see Grace Hopper white paper) using nvlink, never touching the PCIe bus...
    • shaklee3 4y ago
      the problem is PCIe speeds are increasing faster than network. pcie5 can hit 500Gbps in theory, but the network is only 400Gbps.