Outsider speculation, but I’d guess ROI and compute/infra limitations play a big factor here. LLMs have well understood limitations simply because of their inte

Outsider speculation, but I’d guess ROI and compute/infra limitations play a big factor here. LLMs have well understood limitations simply because of their internal machinery - they’re probabilistic, can’t dynamically learn (arguable), stateless. Every major AI lab is working on new techniques to address some of these limitations. If we just decided to charge ahead and keep upping the parameter counts of these models by orders of magnitude, we’d probably see big improvements in the quality of the models, but is it worth it? Inferencing even at current model sizes is already notoriously expensive, and training is also both expensive, resource constrained (see OpenAI recently signing the deal with Oracle for more DCs) and technically difficult because you’re now doing training across entire data centres. It seems reasonable for these labs to be focusing on fundamental improvements and research, while making incremental improvements to existing models at least until the compute catches up and it becomes economically feasible to just jam up parameter counts again.