Then you're including all kinds of time for other delays, like copying data over PCIe into the GPU for the next batch, or waiting for the CPU to orchestrate thi

Then you're including all kinds of time for other delays, like copying data over PCIe into the GPU for the next batch, or waiting for the CPU to orchestrate things. You could probably get a better estimate based on the network structure, the number of training epochs, and the size of each batch.