2 ms·

> own mutable copy of the model, which is usually huge It could just be the diff against the main model or similar.

• quickthrower2 3y ago
  But if you have say 50bn weights, and you run backprop, you are going to update most of the weights (except the dropout ones, but which ones drop out changes on every token I think). This means you need 50bn deltas. It might compress, but if you do then you need extra compute to do that.
  • jacquesm 3y ago
    You would do dropout on every epoch of training, not on every token.
    • quickthrower2 3y ago
      I didn't know that, I might look at the NanoGPT code and torch.dropout docs a bit closer then. Thanks!