4 ms·

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total do

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

simonw 1y ago
The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.
- cwillu 1y ago
  The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true: an academic study does not get funded, executed, written up, and published in less time.
  - Ntrails 1y ago
    Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales
    - Filligree 1y ago
      Or you accept that different people have different skill levels, workflows and goals, and therefore the AIs reach usability at different times.
      - rsynnott 1y ago
        The complication is that, as noted in the above paper, _people are bad at self-reporting on whether the magic robot works for them_. Just because someone _believes_ they are more effective using LLMs is not particularly strong evidence that they actually are.
    - steveklabnik 1y ago
      There’s a lot of confounding factors here. For example, you could point to any of these things in the last ~8 months as being significant changes: * the release of agentic workflow tools * the release of MCPs * the release of new models, Claude 4 and Gemini 2.5 in particular * subagents * asynchronous agents All or any of these could have made for a big or small impact. For example, I’m big on agentic tools, skeptical of MCPs, and don’t think we yet understand subagents. That’s different from those who, for example, think MCPs are the future. > At some point, you roll your eyes and assume it is just snake oil sales No, you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time. There are surely snake oil salesman, but you can’t buy anything from me.
      - Ntrails 1y ago
        > you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time. I pointed this out in my post for a reason. I get it. But even given a different person is saying the same thing every time a new release comes out - the effect on my prior is the same.
- foobarqux 1y ago
  That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before; except that that is the same thing the same people say for every model release, including at the time or release of the previous one, which is now acknowledged to be seriously flawed; and including the future one, at which time the current models will similarly be acknowledged to be, not only less performant that the future models, but inherently flawed. Of course it's possible that at some point you get to a model that really works, irrespective of the history of false claims from the zealots, but it does mean you should take their comments with a grain of salt.
  - steveklabnik 1y ago
    > That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before Right. > except that that is the same thing the same people say for every model release, I did not say that, no. I am sure you can find someone who is in a Groundhog Day about this, but it’s just simpler than that: as tools improve, more people find them useful than before. You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed.
    - blibble 1y ago
      > You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed. no, it's the same names, again and again
      - simonw 1y ago
        Got receipts? That sounds like a claim you could back up with a little bit of time spent using Hacker News search or similar. (I might try to get a tool like o3 to run those searches for me.)
        blibble 1y ago
        try asking it what sealioning is
pdabbadabba 1y ago
Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?
- bryanrasmussen 1y ago
  sure, but after having spent some time trying to get anything useful - programmatically - out of previous models and not getting anything once a new one is announced how much time should one spend. Sure you may end up missing out on a good thing and then having to come late to the party, but coming early to the party too many times and the beer is watered down and the food has grubs is apt to make you cynical the next time a party announcement comes your way.
  - Terr_ 1y ago
    Plus it's not even possible to miss the metaphorical party: If it gets going, it will be quite obvious long before it peaks. (Unless one believes the most grandiose prophecies of a technological-singularity apocalypse, that is.)
- Terr_ 1y ago
  That's not the issue. Their complaint is that proponents keep revising what ought to be fixed goalposts... Well, fixed unless you believe unassisted human developers are also getting dramatically better at their jobs every year. Like the boy who cried wolf, it'll eventually be true with enough time... But we should stop giving them the benefit of the doubt. _____ Jan 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Feb 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Mar 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Apr 2025: [Ad nauseam, you get the idea]
  - pdabbadabba 1y ago
steveklabnik 1y ago
Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.
- mattmanser 1y ago
  Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model comes out you get a bunch of people raving how great the new one is and I honestly can't really tell the difference. The only real difference is reasoning models actually slowed everything down, but now I see its reasoning. It's only useful because I often spot it leaving out important stuff from the final answer.
  - hombre_fatal 1y ago
    I see a massive jump every time. Just two years ago, this failed. > Me: What language is this: "esto está escrito en inglés" > LLM: English Gemini and Opus have solved questions that took me weeks to solve myself. And I'll feed some complex code into each new iteration and it will catch a race condition I missed even with testing and line by line scrutiny. Consider how many more years of experience you need as a software engineer to catch hard race conditions just from reading code than someone who couldn't do it after trying 100 times. We take it for granted already since we see it as "it caught it or it didn't", but these are massive jumps in capability.
  - steveklabnik 1y ago
    Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.
jstummbillig 1y ago
Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New things improve, quickly. That does not mean that something is good or valuable but it's how new tech gets introduced every single time, and readily explains changing sentiment.
- card_zero 1y ago
  I saw that edit. Indeed you can't predict that rejecting a new thing is part of a routine of being wrong. It's true that "it's strange and new, therefore I hate it" is a very human (and adorable) instinct, but sometimes it's reasonable.
  - jstummbillig 1y ago
    "I saw that edit" lol
    - card_zero 1y ago
      Sorry, just happened to. Slightly rude of me.
      - jstummbillig 1y ago
        Ah, you do you. It's just a fairly kindergarten thing to point out and not something I was actively trying to hide. Whatever it was. Generally, I do a couple of edits for clarity after posting and reading again. Sometimes that involves removing something that I feel could have been said better. If it does not work, I will just delete the comment. Whatever it was must not have been a super huge deal (to me).
        maxbond 1y ago
cfst 1y ago
The current batch of models, specifically Claude Sonnet and Opus 4, are the first I've used that have actually been more helpful than annoying on the large mixed-language codebases I work in. I suspect that dividing line differs greatly between developers and applications.
nalllar 1y ago
If you interact with internet comments and discussions as an amorphous blob of people you'll see a constant trickle of the view that models now are useful, and before were useless. If you pay attention to who says it, you'll find that people have different personal thresholds for finding llms useful, not that any given person like steveklabnik above keeps flip-flopping on their view. This is a variant on the goomba fallacy: https://englishinprogress.net/gen-z-slang/goomba-fallacy-explained/ https://englishinprogress.net/gen-z-slang/goomba-fallacy-exp...
bix6 1y ago
Everything actually got better. Look at the image generation improvements as an easily visible benchmark. I do not program for my day job and I vibe coded two different web projects. One in twenty mins as a test with cloudflare deployment having never used cloudflare and one in a week over vacation (and then fixed a deep safari bug two weeks later by hammering the LLM). These tools massively raise the capabilities for sub-average people like me and decrease the time / brain requirements significantly. I had to make a little update to reset the KV store on cloudflare and the LLM did it in 20s after failing the syntax twice. I would’ve spent at least a few minutes looking it up otherwise.
Aeolun 1y ago
It’s true though? Previous models could do well in specifically created settings. You can throw practically everything at Opus, and it’ll work mostly fine.
mwigdahl 1y ago
I've been a proponent for a long time, so I certainly fit this at least partially. However, the combination of Claude Code and the Claude 4 models has pushed the response to my demos of AI coding at my org from "hey, that's kind of cool" to "Wow, can you get me an API key please?" It's been a very noticeable uptick in power, and although there have been some nice increases with past model releases, this has been both the largest and the one that has unlocked the most real value since I've been following the tech.
- achierius 1y ago
  Is that really the case vs. 3.7? For me that was the threshold, and since then the improvements have been nice but not as significant.
  - mwigdahl 1y ago
    I would agree with you that the jump from Sonnet 3.7 to Sonnet 4 feels notable but not shocking. Opus 4 is considerably better, and Opus 4 combined with the Claude Code harness is what really unlocks the value for me.
itsoktocry 1y ago
>the previous model retroactively becomes total dogshit the moment a new one is released Keep writing your code manually, nobody cares.
- player1234 1y ago
  And nobody will notice.

simonw 1y ago

The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.

cwillu 1y ago
The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true: an academic study does not get funded, executed, written up, and published in less time.
- Ntrails 1y ago
  Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales
  - Filligree 1y ago
    Or you accept that different people have different skill levels, workflows and goals, and therefore the AIs reach usability at different times.
    - rsynnott 1y ago
      The complication is that, as noted in the above paper, _people are bad at self-reporting on whether the magic robot works for them_. Just because someone _believes_ they are more effective using LLMs is not particularly strong evidence that they actually are.
  - steveklabnik 1y ago
    There’s a lot of confounding factors here. For example, you could point to any of these things in the last ~8 months as being significant changes: * the release of agentic workflow tools * the release of MCPs * the release of new models, Claude 4 and Gemini 2.5 in particular * subagents * asynchronous agents All or any of these could have made for a big or small impact. For example, I’m big on agentic tools, skeptical of MCPs, and don’t think we yet understand subagents. That’s different from those who, for example, think MCPs are the future. > At some point, you roll your eyes and assume it is just snake oil sales No, you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time. There are surely snake oil salesman, but you can’t buy anything from me.
    - Ntrails 1y ago
      > you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time. I pointed this out in my post for a reason. I get it. But even given a different person is saying the same thing every time a new release comes out - the effect on my prior is the same.
foobarqux 1y ago
That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before; except that that is the same thing the same people say for every model release, including at the time or release of the previous one, which is now acknowledged to be seriously flawed; and including the future one, at which time the current models will similarly be acknowledged to be, not only less performant that the future models, but inherently flawed. Of course it's possible that at some point you get to a model that really works, irrespective of the history of false claims from the zealots, but it does mean you should take their comments with a grain of salt.
- steveklabnik 1y ago
  > That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before Right. > except that that is the same thing the same people say for every model release, I did not say that, no. I am sure you can find someone who is in a Groundhog Day about this, but it’s just simpler than that: as tools improve, more people find them useful than before. You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed.
  - blibble 1y ago
    > You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed. no, it's the same names, again and again
    - simonw 1y ago
      Got receipts? That sounds like a claim you could back up with a little bit of time spent using Hacker News search or similar. (I might try to get a tool like o3 to run those searches for me.)
      - blibble 1y ago
        try asking it what sealioning is

pdabbadabba 1y ago

Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?

bryanrasmussen 1y ago
sure, but after having spent some time trying to get anything useful - programmatically - out of previous models and not getting anything once a new one is announced how much time should one spend. Sure you may end up missing out on a good thing and then having to come late to the party, but coming early to the party too many times and the beer is watered down and the food has grubs is apt to make you cynical the next time a party announcement comes your way.
- Terr_ 1y ago
  Plus it's not even possible to miss the metaphorical party: If it gets going, it will be quite obvious long before it peaks. (Unless one believes the most grandiose prophecies of a technological-singularity apocalypse, that is.)
Terr_ 1y ago
That's not the issue. Their complaint is that proponents keep revising what ought to be fixed goalposts... Well, fixed unless you believe unassisted human developers are also getting dramatically better at their jobs every year. Like the boy who cried wolf, it'll eventually be true with enough time... But we should stop giving them the benefit of the doubt. _____ Jan 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Feb 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Mar 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious." Apr 2025: [Ad nauseam, you get the idea]
- pdabbadabba 1y ago

steveklabnik 1y ago

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

mattmanser 1y ago
Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model comes out you get a bunch of people raving how great the new one is and I honestly can't really tell the difference. The only real difference is reasoning models actually slowed everything down, but now I see its reasoning. It's only useful because I often spot it leaving out important stuff from the final answer.
- hombre_fatal 1y ago
  I see a massive jump every time. Just two years ago, this failed. > Me: What language is this: "esto está escrito en inglés" > LLM: English Gemini and Opus have solved questions that took me weeks to solve myself. And I'll feed some complex code into each new iteration and it will catch a race condition I missed even with testing and line by line scrutiny. Consider how many more years of experience you need as a software engineer to catch hard race conditions just from reading code than someone who couldn't do it after trying 100 times. We take it for granted already since we see it as "it caught it or it didn't", but these are massive jumps in capability.
- steveklabnik 1y ago
  Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.

jstummbillig 1y ago

Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New things improve, quickly. That does not mean that something is good or valuable but it's how new tech gets introduced every single time, and readily explains changing sentiment.

card_zero 1y ago
I saw that edit. Indeed you can't predict that rejecting a new thing is part of a routine of being wrong. It's true that "it's strange and new, therefore I hate it" is a very human (and adorable) instinct, but sometimes it's reasonable.
- jstummbillig 1y ago
  "I saw that edit" lol
  - card_zero 1y ago
    Sorry, just happened to. Slightly rude of me.
    - jstummbillig 1y ago
      Ah, you do you. It's just a fairly kindergarten thing to point out and not something I was actively trying to hide. Whatever it was. Generally, I do a couple of edits for clarity after posting and reading again. Sometimes that involves removing something that I feel could have been said better. If it does not work, I will just delete the comment. Whatever it was must not have been a super huge deal (to me).
      - maxbond 1y ago

cfst 1y ago

The current batch of models, specifically Claude Sonnet and Opus 4, are the first I've used that have actually been more helpful than annoying on the large mixed-language codebases I work in. I suspect that dividing line differs greatly between developers and applications.

nalllar 1y ago

If you interact with internet comments and discussions as an amorphous blob of people you'll see a constant trickle of the view that models now are useful, and before were useless. If you pay attention to who says it, you'll find that people have different personal thresholds for finding llms useful, not that any given person like steveklabnik above keeps flip-flopping on their view. This is a variant on the goomba fallacy: https://englishinprogress.net/gen-z-slang/goomba-fallacy-explained/ https://englishinprogress.net/gen-z-slang/goomba-fallacy-exp...

bix6 1y ago

Everything actually got better. Look at the image generation improvements as an easily visible benchmark. I do not program for my day job and I vibe coded two different web projects. One in twenty mins as a test with cloudflare deployment having never used cloudflare and one in a week over vacation (and then fixed a deep safari bug two weeks later by hammering the LLM). These tools massively raise the capabilities for sub-average people like me and decrease the time / brain requirements significantly. I had to make a little update to reset the KV store on cloudflare and the LLM did it in 20s after failing the syntax twice. I would’ve spent at least a few minutes looking it up otherwise.

Aeolun 1y ago

It’s true though? Previous models could do well in specifically created settings. You can throw practically everything at Opus, and it’ll work mostly fine.

mwigdahl 1y ago

I've been a proponent for a long time, so I certainly fit this at least partially. However, the combination of Claude Code and the Claude 4 models has pushed the response to my demos of AI coding at my org from "hey, that's kind of cool" to "Wow, can you get me an API key please?" It's been a very noticeable uptick in power, and although there have been some nice increases with past model releases, this has been both the largest and the one that has unlocked the most real value since I've been following the tech.

achierius 1y ago
Is that really the case vs. 3.7? For me that was the threshold, and since then the improvements have been nice but not as significant.
- mwigdahl 1y ago
  I would agree with you that the jump from Sonnet 3.7 to Sonnet 4 feels notable but not shocking. Opus 4 is considerably better, and Opus 4 combined with the Claude Code harness is what really unlocks the value for me.

itsoktocry 1y ago

>the previous model retroactively becomes total dogshit the moment a new one is released Keep writing your code manually, nobody cares.

player1234 1y ago
And nobody will notice.