3 ms·
> So, these models are basically within 1% of each other in terms of final pretraining loss. How is this loss calculated though? Since it is called "loss" and
by radarsat1 2y ago
> So, these models are basically within 1% of each other in terms of final pretraining loss.
How is this loss calculated though? Since it is called "loss" and not "performance metric", I'm going to assume it is the teacher forced cross entropy loss.
I'm not too familiar with LLM training but having been doing some fair amount of seq2seq training lately in other domains I've observed that the relationship between "loss" and autoregressive inference performance gets very narrow towards the end of training. What I mean is that smaller and smaller reductions in loss lead to better improvements in the autoregressive output. So at least I have some doubt that in practice that 1% loss improvement is not actually incredibly significant with respect to how well the model actually performs at inference time.
But I'm pretty interested in this topic and if people here have observed this or the contrary I'd be curious to know.
- ActivatedAI 2y agoPage 9 on the Llama tech report has an interesting graph that predicts task level performance from the cross-entropy loss. The sigmoidal model fits well, and at the steepest part of the S, a .01 change in NLL is worth about 5% task level accuracy. Here is a quick screenshot if you are lazy. https://snipboard.io/C6mipQ.jpg https://snipboard.io/C6mipQ.jpg And here is the paper if you want to dig deep into the highest quality published research on LLM frontier. https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ https://ai.meta.com/research/publications/the-llama-3-herd-o...
- radarsat1 2y agoThanks that's actually pretty much what I was thinking, I'll have to read that to try to understand the significance of the S curve there, pretty interesting. Appreciate the link!