4 ms·
What does overfitting do to a language model
by seydor 3y ago
What does overfitting do to a language model
- samus 3y agoProbably nothing specific. By definition, an overfitted model has only learned to deal with the training input and will perform worse on anything that is different. In training metrics it shows as a large gap between the performance metrics on the training test vs. on the test set.
- exe34 3y agoWhen the training data is the entire internet, I suspect it can still perform well enough in economically profitable situations to be worth having.
- samus 3y agoSurely it might still work, but training a model into true overfitting territory is a waste of resources. An overtrained model is usually a worse model than an earlier checkpoint. That's the main reason one strives to avoid that.
- fancyfredbot 3y agoAn overfit LLM would probably regurgitate common phrases from the training dataset and struggle to output phrases which didn't occur in that dataset. I'm not really sure what that looks like when your training data is so huge. Might not be very noticeable?
- seydor 3y agoalso, an LLM is not learning phrases per se but the relationships between tokens