4 ms·

It's weird to see people claiming that model capabilities are plateauing. It wasn't until late last year that we even had strong coding models. Imagine if, less

It's weird to see people claiming that model capabilities are plateauing. It wasn't until late last year that we even had strong coding models. Imagine if, less than a year after the first iPhone launched, people claimed that smartphone capabilities were "plateauing" because Apple hadn't yet launched a new phone. And it seems the issue is less than "models aren't getting better" than, "models are good enough to handle 99% of the coding tasks people give to them".

rimliu 3mo ago
People claim what they see. I see no improvement since opus 4.6, quite the opossite.
- qtk8 3mo ago
  Which is not even 5 months old.
- usef- 3mo ago
  That's only a few months old. Just because there's time between big releases doesn't mean progress stopped. Fable seemed very clearly a step up in my one afternoon of usage. I gave it several bugs that other models had failed at repeatedly (in a mess of a vibe coded side project) and it fixed them each in one prompt.
  - jijijijij 3mo ago
    > Just because there's time between big releases doesn't mean progress stopped. No, but progress not stopping doesn't mean it's not plateauing. I believe 'plateauing' is understood as the process of approaching a plateau, not being stuck on a plateau already. So, the question is about the rate of progress, not its existence.
    - usef- 3mo ago
      I guess we draw a different line then. This year has been full of a lot of great releases so far. Most normal people didn't even use Agents before January. It does not at all feel slower than previous years. HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release. 6 months ago: > LLMs are amazing, but they have reached a plateau. https://news.ycombinator.com/item?id=46109534 https://news.ycombinator.com/item?id=46109534 1 year ago: > generative AI has languished in the same place, even in my kindest estimations, for several months, though it's really been years. https://news.ycombinator.com/item?id=43085885 https://news.ycombinator.com/item?id=43085885 2 years ago: > 2024 has seen nothing substantially good and the only notesworthy thing is this article finally hitting into the public consciousness that we are past of the AI peak and beyond the plateau and freefalling has already begun. https://news.ycombinator.com/item?id=42125888 https://news.ycombinator.com/item?id=42125888 --- Many more that I haven't time to look up. I think the present just always feels slow.
      - jijijijij 3mo ago
        It doesn't matter where we draw the line, or what someone else said at some point. I merely pointed out your argument wasn't a sound rebuttal. Criticizing subjective experiences to assess the situation is, but that cuts both ways. > HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release. Your earliest example was 2 years after release, when LLMs were already widely used and there is literally a source to support the claim. Now, you need to show research efforts, time and resource investments, ... are producing proportional results, disproving diminishing returns. If there are diminishing returns, LLMs are plateauing. Also, manual capability extensions, or use/edge case adaptations, which may improve subjective usability are not exactly advances in AI as technology. LLMs still hallucinate. LLms still fundamentally struggle with certain classes of problems (e.g. counting), but you increasingly need to come up with different problem dress-ups because of targeted interventions to manufacture hype and a limited supply of test cases. Can you make the case AI got actually more intelligent, fundamentally? That is, not an increase in case specific usability, but a decrease in fundamental limitations. And is this proportional to improvement efforts?
        3mo ago
IsTom 3mo ago
I wonder how's model improvement/dollar invested ratio going. If gains are made by simply spending increasingly vast amounts of cash that's not going to last.

rimliu 3mo ago

People claim what they see. I see no improvement since opus 4.6, quite the opossite.

qtk8 3mo ago
Which is not even 5 months old.
usef- 3mo ago
That's only a few months old. Just because there's time between big releases doesn't mean progress stopped. Fable seemed very clearly a step up in my one afternoon of usage. I gave it several bugs that other models had failed at repeatedly (in a mess of a vibe coded side project) and it fixed them each in one prompt.
- jijijijij 3mo ago
  > Just because there's time between big releases doesn't mean progress stopped. No, but progress not stopping doesn't mean it's not plateauing. I believe 'plateauing' is understood as the process of approaching a plateau, not being stuck on a plateau already. So, the question is about the rate of progress, not its existence.
  - usef- 3mo ago
    I guess we draw a different line then. This year has been full of a lot of great releases so far. Most normal people didn't even use Agents before January. It does not at all feel slower than previous years. HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release. 6 months ago: > LLMs are amazing, but they have reached a plateau. https://news.ycombinator.com/item?id=46109534 https://news.ycombinator.com/item?id=46109534 1 year ago: > generative AI has languished in the same place, even in my kindest estimations, for several months, though it's really been years. https://news.ycombinator.com/item?id=43085885 https://news.ycombinator.com/item?id=43085885 2 years ago: > 2024 has seen nothing substantially good and the only notesworthy thing is this article finally hitting into the public consciousness that we are past of the AI peak and beyond the plateau and freefalling has already begun. https://news.ycombinator.com/item?id=42125888 https://news.ycombinator.com/item?id=42125888 --- Many more that I haven't time to look up. I think the present just always feels slow.
    - jijijijij 3mo ago
      It doesn't matter where we draw the line, or what someone else said at some point. I merely pointed out your argument wasn't a sound rebuttal. Criticizing subjective experiences to assess the situation is, but that cuts both ways. > HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release. Your earliest example was 2 years after release, when LLMs were already widely used and there is literally a source to support the claim. Now, you need to show research efforts, time and resource investments, ... are producing proportional results, disproving diminishing returns. If there are diminishing returns, LLMs are plateauing. Also, manual capability extensions, or use/edge case adaptations, which may improve subjective usability are not exactly advances in AI as technology. LLMs still hallucinate. LLms still fundamentally struggle with certain classes of problems (e.g. counting), but you increasingly need to come up with different problem dress-ups because of targeted interventions to manufacture hype and a limited supply of test cases. Can you make the case AI got actually more intelligent, fundamentally? That is, not an increase in case specific usability, but a decrease in fundamental limitations. And is this proportional to improvement efforts?
      - 3mo ago

IsTom 3mo ago

I wonder how's model improvement/dollar invested ratio going. If gains are made by simply spending increasingly vast amounts of cash that's not going to last.