3 ms·

Have people experimented with distributed training of parts of the model to avoid needing these absolutely massive GPUs? Anyone have pointers to large scale dis

Have people experimented with distributed training of parts of the model to avoid needing these absolutely massive GPUs? Anyone have pointers to large scale distributed training done recently?

fock 3y ago
this is distributed training with RDMA-aware and directly interconnected GPUs?
Aromasin 3y ago
Yes, certainly. One industry use-case that comes to mind if Baidu; white-paper link below [1]. Pretty much all the large model developers distribute their model training across hardware in some way, using a blend of GPU/TPU/FPGA accelerators across multiple CPU nodes. Moving all the data around is expensive though, in both power consumption and time, which is why NVIDIA's new system would be of interest. [1] http://research.baidu.com/Public/uploads/5e76df66c467b.pdf http://research.baidu.com/Public/uploads/5e76df66c467b.pdf
- sour-taste 3y ago
  This is fantastic, thanks.
wongarsu 3y ago
The issue is that distributed training needs high bandwidth and very low latency to be efficient. In a single computer you can fit about 8-10 GPUs, or if you go to extremes like in this system you might fit 16. To scale beyond that, you connect multiple computers in the same rack via Infiniband (a optical fibre network solution, the system in the article comes with a 400G Infiniband network adapter). But systems that can host many GPUs tend to be expensive, and electricity is expensive, so at scale the expensive GPUs make sense. For a homebrew solution you can stick four consumer GPUs in a case and might save a buck.
nojito 3y ago
Yes. OpenAI is a heavy user of Ray.io https://thenewstack.io/how-ray-a-distributed-ai-framework-helps-power-chatgpt/ https://thenewstack.io/how-ray-a-distributed-ai-framework-he...
bogwog 3y ago
There's also Petals: https://petals.ml/ https://petals.ml/
aseipp 3y ago
The DGX described here is a distributed system in the sense that many nodes, each with their own GPUs, exist and are part of the overall whole. They are connected over Infiniband and use RDMA in order to read/write memory across the cluster. Therefore training is also distributed among the nodes in the sense that each node takes part of the process. The difference is that Nvidia's software and hardware stack combined makes all these systems, all these aggregate GPUs, look like One Really Big GPU. Not hundreds of small ones. That's not only good for users because they can take existing programs and migrate them to these big machines and get improved performance, but also good because it's generally much easier to program "one big machine" as opposed to programming and orchestrating many small ones. This is an attractive proposition for many but it requires an insane amount of integration to achieve. So, the major differentiator here isn't the lack of or existence of many discrete machines connected together. It's the programming model, at this scale, that's different. And Nvidia is way ahead of everyone else here in terms of programming models; once full heterogeneous memory management for CUDA arrives in a stable consumer driver, it'll be a massive change for others to catch up with. What you might also be referring to is the idea of "distributed training", or what is called "ensemble learning" where you individually train a bunch of small unique models that, when combined together, perform better than if they were one giant model (or at least are as accurate/efficient as a giant model.) It's "The P2P model" of training because you can take lots of small models and collectively aggregate them. That's an open problem people are attacking but not really relevant in the case of the DGX. Many hyperscalers, such as Microsoft and their project "Brainwave", have very complex heterogeneous AI datacenter stacks consisting of GPUs, FPGAs, TPUs and CPUs. (Google "Microsoft Brainwave" for some papers.) This DGX is positioned as an alternative to that but also as a tool for their customers to use since many want to train large models efficiently.
- LtdJorge 3y ago
  It's not InfiniBand, they're using NVlink thid time
  - High_Tech 3y ago
    NVLink is the GPU interconnect and part of the memory fabric. InfiniBand at 400Gbps is the data and storage interconnect for the CPUs but also to build clusters bigger than 256 GPUs. It's all in the docs.

Have people experimented with distributed training of parts of the model to avoid needing these absolutely massive GPUs? Anyone have pointers to large scale distributed training done recently?

fock 3y ago

this is distributed training with RDMA-aware and directly interconnected GPUs?

Aromasin 3y ago

Yes, certainly. One industry use-case that comes to mind if Baidu; white-paper link below [1]. Pretty much all the large model developers distribute their model training across hardware in some way, using a blend of GPU/TPU/FPGA accelerators across multiple CPU nodes. Moving all the data around is expensive though, in both power consumption and time, which is why NVIDIA's new system would be of interest. [1] http://research.baidu.com/Public/uploads/5e76df66c467b.pdf http://research.baidu.com/Public/uploads/5e76df66c467b.pdf

sour-taste 3y ago
This is fantastic, thanks.

wongarsu 3y ago

The issue is that distributed training needs high bandwidth and very low latency to be efficient. In a single computer you can fit about 8-10 GPUs, or if you go to extremes like in this system you might fit 16. To scale beyond that, you connect multiple computers in the same rack via Infiniband (a optical fibre network solution, the system in the article comes with a 400G Infiniband network adapter). But systems that can host many GPUs tend to be expensive, and electricity is expensive, so at scale the expensive GPUs make sense. For a homebrew solution you can stick four consumer GPUs in a case and might save a buck.

nojito 3y ago

Yes. OpenAI is a heavy user of Ray.io https://thenewstack.io/how-ray-a-distributed-ai-framework-helps-power-chatgpt/ https://thenewstack.io/how-ray-a-distributed-ai-framework-he...

bogwog 3y ago

There's also Petals: https://petals.ml/ https://petals.ml/

aseipp 3y ago

The DGX described here is a distributed system in the sense that many nodes, each with their own GPUs, exist and are part of the overall whole. They are connected over Infiniband and use RDMA in order to read/write memory across the cluster. Therefore training is also distributed among the nodes in the sense that each node takes part of the process. The difference is that Nvidia's software and hardware stack combined makes all these systems, all these aggregate GPUs, look like One Really Big GPU. Not hundreds of small ones. That's not only good for users because they can take existing programs and migrate them to these big machines and get improved performance, but also good because it's generally much easier to program "one big machine" as opposed to programming and orchestrating many small ones. This is an attractive proposition for many but it requires an insane amount of integration to achieve. So, the major differentiator here isn't the lack of or existence of many discrete machines connected together. It's the programming model, at this scale, that's different. And Nvidia is way ahead of everyone else here in terms of programming models; once full heterogeneous memory management for CUDA arrives in a stable consumer driver, it'll be a massive change for others to catch up with. What you might also be referring to is the idea of "distributed training", or what is called "ensemble learning" where you individually train a bunch of small unique models that, when combined together, perform better than if they were one giant model (or at least are as accurate/efficient as a giant model.) It's "The P2P model" of training because you can take lots of small models and collectively aggregate them. That's an open problem people are attacking but not really relevant in the case of the DGX. Many hyperscalers, such as Microsoft and their project "Brainwave", have very complex heterogeneous AI datacenter stacks consisting of GPUs, FPGAs, TPUs and CPUs. (Google "Microsoft Brainwave" for some papers.) This DGX is positioned as an alternative to that but also as a tool for their customers to use since many want to train large models efficiently.

LtdJorge 3y ago
It's not InfiniBand, they're using NVlink thid time
- High_Tech 3y ago
  NVLink is the GPU interconnect and part of the memory fabric. InfiniBand at 400Gbps is the data and storage interconnect for the CPUs but also to build clusters bigger than 256 GPUs. It's all in the docs.