It’s basically linear as long as CPU can feed the GPU with data. That’s because there’s no cross-GPU communication during the expensive part of the computation

It’s basically linear as long as CPU can feed the GPU with data. That’s because there’s no cross-GPU communication during the expensive part of the computation (ie forward and backward pass of a neural network). You shove data in, GPU computes gradient, you apply weight updates and redistribute new weights (which are usually not that large). Running the same workload on even the highest end general purpose CPU is glacial compared to even a modestly priced Pascal GPU. That said, there is a limitation: you can’t really have more than 16GB per GPU today, no matter how much you pay, and GPUs can’t do branchy workloads like SQL, compression, etc.