For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bu

For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bus is no bottleneck. I.e. copying a minibatch of float images is still plenty fast because the gradients / SGD iteration is so slow and requires so much compute (even with mixed precision). For shallower networks, there can be an advantage to copying just the original compressed data to GPU memory and then doing decompression etc there. But modern GPUs haven’t adopted pcie 5 yet because the raw compute is more important. Lastly tensor cores have had a big impact, depending on the network they can be fast enough to be very under-utilized.