4 ms·

The most impressive part about the dgx2 is the nvswitch architecture. Nvlink was a stepping stone to get there, and nvswitch is the next progression to make a u

The most impressive part about the dgx2 is the nvswitch architecture. Nvlink was a stepping stone to get there, and nvswitch is the next progression to make a unified address space on up to 16 GPUs across 2 boxes. I think the biggest threat for Nvidia competitors will be switch manufacturers. The biggest bottleneck continues to be the interconnects between cards and nodes. I wouldn't be surprised if Nvidia designs a proprietary external switch that can connect many dgx-2 as the next step, reducing the need for Ethernet or infiniband. I think the impact of nvswitch cannot be understated, and I don't think it gets much press.

arnon 8y ago
NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.
- shaklee3 8y ago
  Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.
  - arnon 8y ago
    RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
    - shaklee3 8y ago
      It doesn't matter if it bypasses pcie. The fastest available ethernet interface today is 100Gbps. Pcie gen3 x16 will get you to about 115Gbps, so the limiter is the interface speed. Pcie4 should be common when 200G interfaces come out. This is why an external nvswitch can make network switches vulnerable in this market. Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.
    - twtw 8y ago
      This is true, but it is essentially the same situation as using the CPU for processing.
    - janfrode 8y ago
      There’s actually six 50 GB/s nvlinks on each Power9 cpu (300GB/s), 2 or 3 to each GPU, depending on if you have a 4 or 6 GPU system. I’ve seen speculation that the DGX-2 nvswitches has enough free ports to fully connect in 2 Power9 cpus with 300 GB/s each, that would be an interesting system..
m_mueller 8y ago
Exactly, that was my point with the tree of NVswitches. That's the obvious next progression. But to have an interconnect that fast, as I wrote, is also a change in mindset in how you can treat that system. A unified address space is no fun if only a small fraction of the memory is available with high throughput on a specific card, so you are essentially back to a version of cache locality optimizations that are really painful to program for in anything that's not a simple parallel data layout. Having that bandwidth to the whole DGX allows entirely new use usecases.
- shaklee3 8y ago
  Yup. There was a great talk at the last GPU tech conference in san jose where a guy from nvidia showed benchmarks of local memory accesses compared to remote (another GPU over nvlink/nvswitch). There was almost no difference once you got to a large enough transfer size, which is the most common. And the beauty was the only difference is the pointer happened to be on another GPU rather than yours. No special programming was required. Of course, this locks you into Nvidia, but their intent is clear with CUDA already.

arnon 8y ago

NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.

shaklee3 8y ago
Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.
- arnon 8y ago
  RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
  - shaklee3 8y ago
    It doesn't matter if it bypasses pcie. The fastest available ethernet interface today is 100Gbps. Pcie gen3 x16 will get you to about 115Gbps, so the limiter is the interface speed. Pcie4 should be common when 200G interfaces come out. This is why an external nvswitch can make network switches vulnerable in this market. Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.
  - twtw 8y ago
    This is true, but it is essentially the same situation as using the CPU for processing.
  - janfrode 8y ago
    There’s actually six 50 GB/s nvlinks on each Power9 cpu (300GB/s), 2 or 3 to each GPU, depending on if you have a 4 or 6 GPU system. I’ve seen speculation that the DGX-2 nvswitches has enough free ports to fully connect in 2 Power9 cpus with 300 GB/s each, that would be an interesting system..

m_mueller 8y ago

Exactly, that was my point with the tree of NVswitches. That's the obvious next progression. But to have an interconnect that fast, as I wrote, is also a change in mindset in how you can treat that system. A unified address space is no fun if only a small fraction of the memory is available with high throughput on a specific card, so you are essentially back to a version of cache locality optimizations that are really painful to program for in anything that's not a simple parallel data layout. Having that bandwidth to the whole DGX allows entirely new use usecases.

shaklee3 8y ago
Yup. There was a great talk at the last GPU tech conference in san jose where a guy from nvidia showed benchmarks of local memory accesses compared to remote (another GPU over nvlink/nvswitch). There was almost no difference once you got to a large enough transfer size, which is the most common. And the beauty was the only difference is the pointer happened to be on another GPU rather than yours. No special programming was required. Of course, this locks you into Nvidia, but their intent is clear with CUDA already.