4 ms·

Yeah. FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V1

Yeah. FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V100s are super rare. Usually for GPGPU applications the CPU is mostly idle, and only "orchestrates" the GPU usage. If one needs to support both GPGPU-only and CPU-only applications in a cluster, one would usually just build two separate partitions. One with e.g. 4-8 V100s per node, but way weaker and cheaper CPUs, and the other one with 2 beefy CPU sockets but no GPUs. Also, if you are going for that kind of interconnects, you are probably building a system with >100 nodes. For your price, that's 160 million dollars, which is on the ballpark of what these systems costs. You usually get a cheaper price per node at that scale (although not much cheaper), but there are many costs (installation, maintenance, training, etc.) that are not covered there. In particular, those nodes will be running quite hot (just add the watts that those V100s and the CPUs use!), so the cooling infrastructure required to held those in a relatively small space can also end up costing quite a bit. You don't want a water pipe breaking and destroying a rack containing 10 of these nodes costing 1.6 million dollars, right?

dekhn 6y ago
On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. As for your comments on water pipes breaking: I work on Google's TPUv3, which uses water cooling to the TPU chips, see https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/ https://www.nextplatform.com/2018/05/10/tearing-apart-google... for some speculation on the details.
- fluffything 6y ago
  > On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. This depends on both the application and the actual machine. Moving data from storage or from the network to the CPU to then feed it to the GPU just increases latency. Unless you are doing some non-parallelizable pre-processing of the data in the CPU, you are often better off with doing DMA or RDMA directly to the GPU memory, fully bypassing the CPU. Not all machines support that, and not all applications support that either, but many applications do. Arguably, even for the applications that do this, leaving the CPUs unused is not really a desirable property. I've seen a couple of applications that split the compute workload to the CPU to fully utilize the CPU as well, and I've also seem other applications that once the GPU finishes processing they move the data to the CPU memory to leave space on the GPU for the next batch as quickly as possible, and the CPU then streams the data off the node while the GPU continues to compute (so you get an RDMA->GPU->CPU->Storage/Network loop). > see https://www.nextplatform.com/2018/05/10/tearing-apart-google.. https://www.nextplatform.com/2018/05/10/tearing-apart-google.... for some speculation on the details. Thanks for the link, I never learned much about this part of the system.

dekhn 6y ago

On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. As for your comments on water pipes breaking: I work on Google's TPUv3, which uses water cooling to the TPU chips, see https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/ https://www.nextplatform.com/2018/05/10/tearing-apart-google... for some speculation on the details.

fluffything 6y ago
> On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. This depends on both the application and the actual machine. Moving data from storage or from the network to the CPU to then feed it to the GPU just increases latency. Unless you are doing some non-parallelizable pre-processing of the data in the CPU, you are often better off with doing DMA or RDMA directly to the GPU memory, fully bypassing the CPU. Not all machines support that, and not all applications support that either, but many applications do. Arguably, even for the applications that do this, leaving the CPUs unused is not really a desirable property. I've seen a couple of applications that split the compute workload to the CPU to fully utilize the CPU as well, and I've also seem other applications that once the GPU finishes processing they move the data to the CPU memory to leave space on the GPU for the next batch as quickly as possible, and the CPU then streams the data off the node while the GPU continues to compute (so you get an RDMA->GPU->CPU->Storage/Network loop). > see https://www.nextplatform.com/2018/05/10/tearing-apart-google.. https://www.nextplatform.com/2018/05/10/tearing-apart-google.... for some speculation on the details. Thanks for the link, I never learned much about this part of the system.