The word scaling has multiple definitions. It's not just the same hardware getting cheaper over time, there have been many architectural improvements: - Bett

The word scaling has multiple definitions. It's not just the same hardware getting cheaper over time, there have been many architectural improvements: - Better training algorithms for training with lower precision - fastattention, fastattention 2 for decreasing bandwidth inside the GPU - ring attention for decreasing bandwidth across GPUs - algorithmic improvements in handling long context window - MoE - Lots of algorithmic improvements in fine tuning / alignment - GROQ hardware architecture (deterministic hardware, storing all data in SRAM for inferencing instead of using cache hierarchies) - Improvements in tokenization So far softmax(K * Q')*V is the only thing that hasn't (yet) been touched. If the question is that ,,just'' improving LLM perplexity by further algorithmic and hardware improvements will lead to AGI, at this point many researchers believe that the answer is yes (and many others that the answer is no :) ).