> On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast

> On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. This depends on both the application and the actual machine. Moving data from storage or from the network to the CPU to then feed it to the GPU just increases latency. Unless you are doing some non-parallelizable pre-processing of the data in the CPU, you are often better off with doing DMA or RDMA directly to the GPU memory, fully bypassing the CPU. Not all machines support that, and not all applications support that either, but many applications do. Arguably, even for the applications that do this, leaving the CPUs unused is not really a desirable property. I've seen a couple of applications that split the compute workload to the CPU to fully utilize the CPU as well, and I've also seem other applications that once the GPU finishes processing they move the data to the CPU memory to leave space on the GPU for the next batch as quickly as possible, and the CPU then streams the data off the node while the GPU continues to compute (so you get an RDMA->GPU->CPU->Storage/Network loop). > see https://www.nextplatform.com/2018/05/10/tearing-apart-google.. https://www.nextplatform.com/2018/05/10/tearing-apart-google.... for some speculation on the details. Thanks for the link, I never learned much about this part of the system.