and this is why the LLM arms race for ultra high parameter count models will stagnate. It's all well and good that we're developing interesting new models. But

and this is why the LLM arms race for ultra high parameter count models will stagnate. It's all well and good that we're developing interesting new models. But once you factor cost into the equation it does severely limit what applications justify the cost. Raw FLOPs may increase each generation but VRAM becomes a limiting factor. And fast VRAM is expensive. I do expect to see incremental innovation in reducing the size of foundational models.