Complicated question, the answer is “it depends”. Several factors influence performance beyond parameter count, notable ones include: training corpus quality,

Complicated question, the answer is “it depends”. Several factors influence performance beyond parameter count, notable ones include: training corpus quality, training flops, and the downstream task. It depends on how much compute you are spending on training and how big of a model you’re talking about. There’s a “minimum” tokens/parameter for increasing size to be effective at improving loss/perplexity, so as you go up in parameters you generally will have to broaden your corpus which may lower it’s quality (e.g. tweets/reddit posts vs books/articles). This effect isn’t as significant at 65B parameters as there is still enough high quality training data but if you’re talking 1T the corpus will (probably, I haven’t tried this/seen this done) by necessity be significantly poorer quality as you will overfit by simply repeating (it’s only beneficial so many times to repeat). As a general rule, when validation loss/perplexity are the same in two models of different sizes downstream performance seems to be also generally the same (this was briefly explored in the PaLM 2 paper by Google) although it doesn’t correlate perfectly. Practically speaking, this translates into a bigger model is better for applications we’re generally talking about. It just may not hold infinitely which we’re starting to see evidence of. It’s definitely not linear though, you can look at some of the OpenLLaMA benchmarks (without getting into the weeds of if current benchmarks are representative) and the accuracy improvements in even 13B is not that significant (noting here all models were trained for the smear number of tokens, so relatively overtraining the smaller models). There are probably some threshold parameter sizes that make a big difference but it’s still being determined. https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf