4 ms·

NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really h

by arnon 8y ago

NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.

shaklee3 8y ago
Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.
- arnon 8y ago
  RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
  - shaklee3 8y ago
    It doesn't matter if it bypasses pcie. The fastest available ethernet interface today is 100Gbps. Pcie gen3 x16 will get you to about 115Gbps, so the limiter is the interface speed. Pcie4 should be common when 200G interfaces come out. This is why an external nvswitch can make network switches vulnerable in this market. Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.
  - twtw 8y ago
    This is true, but it is essentially the same situation as using the CPU for processing.
  - janfrode 8y ago
    There’s actually six 50 GB/s nvlinks on each Power9 cpu (300GB/s), 2 or 3 to each GPU, depending on if you have a 4 or 6 GPU system. I’ve seen speculation that the DGX-2 nvswitches has enough free ports to fully connect in 2 Power9 cpus with 300 GB/s each, that would be an interesting system..

by arnon 8y ago

NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.

shaklee3 8y ago

Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.

arnon 8y ago
RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
- shaklee3 8y ago
  It doesn't matter if it bypasses pcie. The fastest available ethernet interface today is 100Gbps. Pcie gen3 x16 will get you to about 115Gbps, so the limiter is the interface speed. Pcie4 should be common when 200G interfaces come out. This is why an external nvswitch can make network switches vulnerable in this market. Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.
- twtw 8y ago
  This is true, but it is essentially the same situation as using the CPU for processing.
- janfrode 8y ago
  There’s actually six 50 GB/s nvlinks on each Power9 cpu (300GB/s), 2 or 3 to each GPU, depending on if you have a 4 or 6 GPU system. I’ve seen speculation that the DGX-2 nvswitches has enough free ports to fully connect in 2 Power9 cpus with 300 GB/s each, that would be an interesting system..