2 ms·
They don't use the old model, they use EMA of the trained model weights as teacher
by big-chungus4 5mo ago
They don't use the old model, they use EMA of the trained model weights as teacher
- HarHarVeryFunny 5mo agoYeah, and they are not self-distilling in the sense of training on the outputs of the model, but instead operating on the pre-output probability distribution - trying to minimize teacher-student differences. Still, at the end of the day this is a type of SFT, not continual learning in the human/animal sense.