8 ms·

There's something qualitatively new in the DGX machines. GPUs have been great for numerics in HPC, but most workloads that matter in HPC (and where there is a m

There's something qualitatively new in the DGX machines. GPUs have been great for numerics in HPC, but most workloads that matter in HPC (and where there is a market for expensive chips) don't fit on single GPU or even single GPU nodes - until the DGX(2). If they can scale this up (i.e. a tree of NVswitches connecting multiple DGX) it means that suddenly you can treat a whole datacenter worth of hardware as single GPU with a single mode of parallelization in the code. Where conventional HPC today does MPI+OpenMP+Vectorization(unrolling, intrinsics etc.) that only a handful of experts can really program effectively, abstracting this to a single interface of kernel definitions is a huge deal. This will eventually (over the next couple of years) allow "users" of the data created to program these machines directly, i.e. HPC engineers may become less necessary. The actual effective usage of these systems has always been one of the important bottlenecks in HPC.

eismcc 8y ago
Chapel language gets you a bit closer to an easier HPC programming model w/o special hardware.
- bitL 8y ago
  Any good MOOC for this? I have a "poor man's DGX" at home (2x2080Ti with NVLink & 2xTesla K80 in peer mode, 24c CPU, 128GB ECC) and would love to have a unified access to all of this.
  - jdc 8y ago
    GPU programming is still only a proposed feature for Chapel. https://github.com/chapel-lang/chapel/issues/5323 https://github.com/chapel-lang/chapel/issues/5323
    - eismcc 8y ago
      I honestly feel that Chapel’s failure to embrace GPUs earlier hindered its adoption. This was the main reason/Hope I moved to it, but it hasn’t materialized.
marmaduke 8y ago
The DGX is impressive if incremental progress. MPI, OpenMP and SIMD are different APIs where CUDA is one, but their problems still need solving, eg inter-node, intra-node/inter-core, and intra-core/lane. It still takes a domain expert to map a problem into those levels effectively. CUDA automates this for tree of GPUs but only do this in a handful of relatively straightforward problems.
shaklee3 8y ago
The most impressive part about the dgx2 is the nvswitch architecture. Nvlink was a stepping stone to get there, and nvswitch is the next progression to make a unified address space on up to 16 GPUs across 2 boxes. I think the biggest threat for Nvidia competitors will be switch manufacturers. The biggest bottleneck continues to be the interconnects between cards and nodes. I wouldn't be surprised if Nvidia designs a proprietary external switch that can connect many dgx-2 as the next step, reducing the need for Ethernet or infiniband. I think the impact of nvswitch cannot be understated, and I don't think it gets much press.
- arnon 8y ago
  NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.
  - shaklee3 8y ago
    Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.
    - arnon 8y ago
      RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
      - 8y ago

eismcc 8y ago

Chapel language gets you a bit closer to an easier HPC programming model w/o special hardware.

bitL 8y ago
Any good MOOC for this? I have a "poor man's DGX" at home (2x2080Ti with NVLink & 2xTesla K80 in peer mode, 24c CPU, 128GB ECC) and would love to have a unified access to all of this.
- jdc 8y ago
  GPU programming is still only a proposed feature for Chapel. https://github.com/chapel-lang/chapel/issues/5323 https://github.com/chapel-lang/chapel/issues/5323
  - eismcc 8y ago
    I honestly feel that Chapel’s failure to embrace GPUs earlier hindered its adoption. This was the main reason/Hope I moved to it, but it hasn’t materialized.

marmaduke 8y ago

The DGX is impressive if incremental progress. MPI, OpenMP and SIMD are different APIs where CUDA is one, but their problems still need solving, eg inter-node, intra-node/inter-core, and intra-core/lane. It still takes a domain expert to map a problem into those levels effectively. CUDA automates this for tree of GPUs but only do this in a handful of relatively straightforward problems.

shaklee3 8y ago

The most impressive part about the dgx2 is the nvswitch architecture. Nvlink was a stepping stone to get there, and nvswitch is the next progression to make a unified address space on up to 16 GPUs across 2 boxes. I think the biggest threat for Nvidia competitors will be switch manufacturers. The biggest bottleneck continues to be the interconnects between cards and nodes. I wouldn't be surprised if Nvidia designs a proprietary external switch that can connect many dgx-2 as the next step, reducing the need for Ethernet or infiniband. I think the impact of nvswitch cannot be understated, and I don't think it gets much press.

arnon 8y ago
NVLink between CPU and GPU is the key, and only IBM has implemented that with Power8 and Power9. Everything NVSwitch does is between GPUs, and doesn't really help with getting data up to the GPUs.
- shaklee3 8y ago
  Going to the CPU is largely irrelevant in most cases. A lot of algorithms just need to transfer data between GPUs or nodes, and RDMA bypasses the CPU completely. NvLink is nice on POWER9, but it's only one or two links instead of the full 6 a GPU has. What we really need is GPUs and CPUs to support PCIe 4/5. Intel's Cascade Lake Xeons don't even support 4, and it's out out until next year. Power has had it for over a year.
  - arnon 8y ago
    RDMA may bypass the CPU but it doesn't bypass PCIe. It still goes through it, which is slower than NVLink on Power9 (~16GB/s on 16x PCIe 3.0 vs ~150GB/s on NVLink). Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too. This adds up.
    - 8y ago