3 ms·
It's 2 for forward pass, 4 for backward pass (2PD + 4PD = 6PD), but we didn't cover why it's 2 and 4 since it was a very deep rabbit hole. Maybe he'll do a sepa
by FanaHOVA 3y ago
It's 2 for forward pass, 4 for backward pass (2PD + 4PD = 6PD), but we didn't cover why it's 2 and 4 since it was a very deep rabbit hole. Maybe he'll do a separate post for that in the future!
- arugulum 3y agoIf you want a speedrun explanation for how we get to "2": In the limit of model scaling, context size doesn't matter (yes, forget about the quadratic attention), most of the compute is in the linear layers, which boil down to matrix multiplies. Consider a single matrix of size [T,d] multiplied by weight of size [d,d], the compute needed for a matrix multiplication is approximately 2Td^2 (2 coming from multiply + add). Swap T out with D for your whole dataset in tokens, d^2 is the number of parameters in a single linear layer so scale up your model to P, and you've got 2PD. Even shorter: The 2 comes from the multiply-add
- float-trip 3y agoThere's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-language-model-training-3b19c1f025e4 https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...