4 ms·
But if you have say 50bn weights, and you run backprop, you are going to update most of the weights (except the dropout ones, but which ones drop out changes on
by quickthrower2 3y ago
But if you have say 50bn weights, and you run backprop, you are going to update most of the weights (except the dropout ones, but which ones drop out changes on every token I think). This means you need 50bn deltas. It might compress, but if you do then you need extra compute to do that.
- jacquesm 3y agoYou would do dropout on every epoch of training, not on every token.
- quickthrower2 3y agoI didn't know that, I might look at the NanoGPT code and torch.dropout docs a bit closer then. Thanks!