4 ms·
How would you even measure that, with current tooling? There some option for counting float ops in pytorch or something?
by jakobson14 3y ago
How would you even measure that, with current tooling? There some option for counting float ops in pytorch or something?
- ketzo 3y agoTime to train * flops per chip * number of chips in training fleet You can estimate it quite accurately, actually
- jakobson14 3y agoThen you're including all kinds of time for other delays, like copying data over PCIe into the GPU for the next batch, or waiting for the CPU to orchestrate things. You could probably get a better estimate based on the network structure, the number of training epochs, and the size of each batch.