4 ms·
Is this method used during training? Seems to me there could be a point to only backpropagate when the model is wrong?
by tednoob 2y ago
Is this method used during training? Seems to me there could be a point to only backpropagate when the model is wrong?
- zwaps 2y agoI mean this is implicit in back propagation, say, you need to store gradients anyway but if you get to a zero loss than you are just done.
- leodriesch 2y agoThe model is always wrong, since it predicts a propability distribution over all possible tokens, but the target has 100% possibility for one token and 0 for all others.