Yes you’re 100% correct, but if you’re using a well tuned GPU Deep Learning pipeline the bottleneck is GPU compute and the CPU is always waiting on the GPU. It

Yes you’re 100% correct, but if you’re using a well tuned GPU Deep Learning pipeline the bottleneck is GPU compute and the CPU is always waiting on the GPU. It can be a bit tricky to optimise this though - some deep learning users do things like resizing images on the fly, or on the fly vectorising / scaling of input data in these cases the CPU can be the bottleneck and the optimisations above can help - but this is largely a symptom of a poor pipeline configuration in newbie shops, most serious places have optimised this away and the GPU is the bottleneck