4 ms·
Gradient decent doesn't per se, but retraining ("fine-tuning") on another dataset forgets most of the training done on the first dataset.
by saiojd 6y ago
Gradient decent doesn't per se, but retraining ("fine-tuning") on another dataset forgets most of the training done on the first dataset.
- joshgel 6y agoIs this true? My understanding was that in fine tuning, you’d only re train some of the layers. And even if you re train all the layers, the starting point for the layers is not random. If it really was all forgotten then fine tuning would not be orders of magnitude faster...
- thunderbird120 6y agoGradient decent optimizes performance of a model on a given dataset. If you stop training on one dataset and start training on another one your model will become more optimized for the second dataset and less optimized for the first. This will usually result in degraded performance on classes of data found more commonly in the first dataset but not the second. This is what people mean by "forgetting". It doesn't matter how much of the model you fine-tune, the effect is still present though the effect size varies.