> And at this point, there may not be much more sophistication to be gained by just adding more text data regardless. There is consensus that almost all contem

> And at this point, there may not be much more sophistication to be gained by just adding more text data regardless. There is consensus that almost all contemporary LLMs are undertrained. See, for example, the Gopher, Chinchilla, and LLaMA papers. Larger models are easier to train, and there are diminishing returns when you keep training. Thus, to claim SotA performance, researchers tried to optimize for the best performance, given a certain training budget. The best performance within a certain training budget is achieved by training HUGE models on VAST datasets, for a relatively short amount of time. Almost all models could profit from simply training for longer, but the cost/benefit isn‘t there, if your goal is to achieve the best performance. This is also why distillation and quantization work so well. Models with more and larger weights are easier to train, but ultimately don‘t utilize all that capacity. Recently, researchers have begun focussing on inference – rather than training – budgets, in order to make using these models actually viable and profitable in practice. I.e., what is the best performance we can achieve within a certain computational limit at inference time? This is mostly done by simply training with more data, for longer.