Not OP, but I'll bite. I, firstly, don't think it's fair to say "they get significantly better every year" after 2 years. I honestly think GPT 3/3.5 (maybe opt

Not OP, but I'll bite. I, firstly, don't think it's fair to say "they get significantly better every year" after 2 years. I honestly think GPT 3/3.5 (maybe optimized for cost or speed but not retrained) would have been adequate for a significant chunk of the general purpose tasks that are asked of LLMs. I think most of the other gains we've seen since are related to fine-tuning on intentional application-specific tasks. IMO the only real "significant" improvement is the native multi-modal models. That said, I think that RL+thinking and long-context models will start to present enough combined incremental improvements towards my next points to be even more useful and capable at a wider variety of tasks, but there is (in their current usable forms like chat bot apps and public APIs) a fundamental limit on their implementation preventing them from being AGI. I think that models, even the "thinking" ones, fail to truly perform novel rationalization around general-purpose task solving. We have a ton of evidence that they're useful for a ton of tasks when it's trained in. Even going a tiny bit in a unique direction drops the model output quality a ton. Available thinking models today are really good at math, because they were RLed into solving math like a school child - rote practice. But that doesn't mean they're capable of even applying the same logical tools (they should have learned along) the way to novel mathematic and logic questions. Another impediment to being treated even as an application-specific "mini AGI" is their naïveté and hallucinations. This makes their use as agents suspect for anything important. They can't distinguish or even output a true confidence on what they "don't know", and this blind confidence is a setback. Humans are known to say incorrect things confidently, but they're also known to reflect on their lack of knowledge and recognize their limits. Humans have real memory to be able (imprecisely) associate an event with their learning, to aid confidence in recall. Similarly, LLMs trusting nature on input (eg. prompt injection attacks) prevent them from "intelligently" acting in the real world even when they're not hallucinating. Tools like "DeepResearch" are really useful, and impressive improvements on traditional human searching for processing the vastness of the internet. BUT the model can't genuinely distinguish between good and bad sources, and often can't intelligently reflect on the patterns and social context of the sources they sell. I can totally see a world where an LLM can output a confidence metric which is used to drive the tokens, and potentially suppress output, and I can totally see a world where long context and thinking (w/ RL) gives it enough reflection on everything to question to function even more autonomously. But I remain skeptical that it will be able to "think" and rationalize deeply enough to be a "super intelligence" on tasks it wasn't taught.