3 ms·
It does start getting worse at some point right?
by Yenrabbit 3y ago
It does start getting worse at some point right?
- jph00 3y agoI'm sure eventually it would, but we haven't gotten to that point yet in our training.
- minihat 3y agoCross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599 Is this 'overconfidence' the leading explanation as to why LLMs continue to show qualitative improvement even after their test loss levels off?
- alekseiprokopev 3y agoIs it possible to somehow modify the sampling from the model to account for that?