3 ms·
LLMs are plateauing. At this point, anyone who has cared enough knows their fundamental limitations. Don’t get me wrong, they do provide immense value and have
by yahoozoo 1y ago
LLMs are plateauing. At this point, anyone who has cared enough knows their fundamental limitations. Don’t get me wrong, they do provide immense value and have grown quickly in just a few years. But, they will probably never get past the “we must treat it like a junior engineer” phase, especially as LLM outputs inevitably leak back into the training data since everyone is using it (or being forced to by employers). Notice that everything major from these companies lately hasn’t been a vastly improved model from the previous iteration, but products and tooling: see Anthropic allowing you to create in-Claude apps, every companies own version of agentic CLI, and so on.
- atemerev 1y agoI don't know. Like, only eight months ago or so, I was explaining my colleagues how using AI for coding can actually work, in some cases. Now, I don't know any engineers who don't use AI tools, at least at work (our own pet projects are still fun to do without AI, but at work, AI-assisted productivity is now expected). If that's "plateauing"...
- nicksergeant 1y agoIt doesn't feel like LLM-assisted coding has gotten materially better in the last 6 months, though. The tooling, sure, but I'm still wrestling with the exact same problems I was at the beginning of the year. That doesn't feel like the exponential improvement people have been prophesizing about.
- atemerev 1y agoI am fine with constant linear improvement too. What model do you use? SOTA is Gemini 2.5-pro. We definitely didn't have anything comparable in 2024.
- yahoozoo 1y agoI wasn’t talking about their adoption. I’m talking about their abilities.
- atemerev 1y agoIn eight months, we got SOTA from 50% to 83% on Aider's Polyglot benchmark (225 Exercism problems). That increase matches my own experience.
- graealex 1y agoOne important thing to note is that everyone has an agenda. Days ago someone wanted to convince me that AI is already outcompeting humans on a broad scale. Yet I go back to ChatGPT and it gives me non-stop hallucinations. So I can only assume that the person is either incompetent and that's why they personally feel that AI is better. Or they already internalized the AI hype so much because they're the ones selling it.
- calebkaiser 1y agoI think this is recency bias. There was roughly 33 months between the initial release of GPT 3 and GPT 4, which was released in March 2023.