2 ms·
Because if you tried to run the supercomputer benchmark on a DC, you'd get a low score, and you can't easily make up for that by adding more computers to a DC.
by dekhn 4y ago
Because if you tried to run the supercomputer benchmark on a DC, you'd get a low score, and you can't easily make up for that by adding more computers to a DC.
To win the supercomputer benchmarks, you need low-latency, high bandwidth networks that allow all the worker nodes in the computer to communicate calculation results. Different real jobs that run on supercomputers have different communications needs but none of them really scale well enough to be economic to run on datacenter style machines.
What's interesting is that over time, the datacenter folks ended up adding supercomputers to their datacenters, with very large and fast database/blob storage/data warehousing systems connected up to "ML supercomputers" (like supercomputers, but typically only do single precision floating point). The two work well together so long as you scale the bandwidth between them. At the end of the day, any interesting data center has obscenely complex networking technology. For example, TPUs are PCI-attached devices in Google data centers; they plug into server machines just like GPUs. The TPUs themselves have networking between TPUs, that allows them to move important data, like gradients, between TPUs, as needed to do gradient descent and other operations, but the hosts that the TPUs are plugged into have their own networks. The TPUs form a mesh- the latest TPUs form a 3D mesh, but physically implemented through a complex optical switch, while the hosts they are attached to multiple switches which themselves from complex graphs of networking elements. When running ML, part of your job might be using the host CPU to read in training data and transform it, keeping the network busy, keeping some remote disk servers busy, while pushing the transformed data into the TPUs, which then communicate internal data between themselves and other TPUs, over an entirely distinct network. Crazy stuff.