You’re not wrong, the Chinchilla rationale is that it may be more compute efficient to obtain a given loss using larger model sizes if the budget allows. As ano

You’re not wrong, the Chinchilla rationale is that it may be more compute efficient to obtain a given loss using larger model sizes if the budget allows. As another commenter states this ignore the inference part of the equation. As an example the BERT/RoBERTa family were trained for much longer than Chinchilla, you do get diminishing returns though. There is a point of overtraining where downstream performance is impacted but that’s pretty high. I think part of the answer to this is also that xxx million parameter decoder-only models don’t seem to be that useful so it may not be worthwhile to optimize them for performance?