The DGX described here is a distributed system in the sense that many nodes, each with their own GPUs, exist and are part of the overall whole. They are connect

The DGX described here is a distributed system in the sense that many nodes, each with their own GPUs, exist and are part of the overall whole. They are connected over Infiniband and use RDMA in order to read/write memory across the cluster. Therefore training is also distributed among the nodes in the sense that each node takes part of the process. The difference is that Nvidia's software and hardware stack combined makes all these systems, all these aggregate GPUs, look like One Really Big GPU. Not hundreds of small ones. That's not only good for users because they can take existing programs and migrate them to these big machines and get improved performance, but also good because it's generally much easier to program "one big machine" as opposed to programming and orchestrating many small ones. This is an attractive proposition for many but it requires an insane amount of integration to achieve. So, the major differentiator here isn't the lack of or existence of many discrete machines connected together. It's the programming model, at this scale, that's different. And Nvidia is way ahead of everyone else here in terms of programming models; once full heterogeneous memory management for CUDA arrives in a stable consumer driver, it'll be a massive change for others to catch up with. What you might also be referring to is the idea of "distributed training", or what is called "ensemble learning" where you individually train a bunch of small unique models that, when combined together, perform better than if they were one giant model (or at least are as accurate/efficient as a giant model.) It's "The P2P model" of training because you can take lots of small models and collectively aggregate them. That's an open problem people are attacking but not really relevant in the case of the DGX. Many hyperscalers, such as Microsoft and their project "Brainwave", have very complex heterogeneous AI datacenter stacks consisting of GPUs, FPGAs, TPUs and CPUs. (Google "Microsoft Brainwave" for some papers.) This DGX is positioned as an alternative to that but also as a tool for their customers to use since many want to train large models efficiently.