3 ms·
You would do dropout on every epoch of training, not on every token.
by jacquesm 3y ago
You would do dropout on every epoch of training, not on every token.
- quickthrower2 3y agoI didn't know that, I might look at the NanoGPT code and torch.dropout docs a bit closer then. Thanks!