3 ms·
I find in the article the assumption almost insulting, that labs haven’t tried training smaller models for longer. Everybody tries that first, of course. It was
by toxik 3y ago
I find in the article the assumption almost insulting, that labs haven’t tried training smaller models for longer. Everybody tries that first, of course. It was the common wisdom for decades. No, it doesn’t work better or even as well. The loss curve flattens out.
- mirekrusin 3y agoOn the other hand models are trained once and later used a lot so you can argue that training cost can be traded for future gains.