33 ms·

I can't be the only one who thinks this version is no better than the previous one, and that LLMs have basically reached a plateau, and all the new releases "fe

• flixing 1y ago
  i think you are.
• go_elmo 1y ago
  I feel like the model making a memory file to store context is more than a gimmick, no?
• brookst 1y ago
  How much have you used Claude 4?
  • hsn915 1y ago
    I asked it a few questions and it responded exactly like all the other models do. Some of the questions were difficult / very specific, and it failed in the same way all the other models failed.
    • theptip 1y ago
      Great example of this general class of reasoning failure. “AI does badly on my test therefore it’s bad”. The correct question to ask is, of course, what is it good at? (For bonus points, think in terms of $/task rather than simply being dominant over humans.)
      • atworkc 1y ago
        "AI does badly on my test much like other AI's did before it, therefore I don't immediately see much improvement" is a fair assumption.
        • brookst 1y ago
          No, it’s really not. “I used an 8088 CPU to whisk egg whites, then an Intel core 9i-12000-vk4*, and they were equally mediocre meringues, therefore the latest Intel processor isn’t a significant improvement over one from 50 years ago” * Bear with me, no idea their current naming
          • Kon-Peki 1y ago
            You’re holding them wrong. An 8088 package should be able to emulate a whisk about a million times better than an i9.
        • theptip 1y ago
          “Human can’t fly, much like other humans. Therefore it’s bad” Spot the problem now? AI capabilities are highly jagged, they are clearly superhuman in many dimensions, and laughably bad compared to humans in others.
• TechDebtDevin 1y ago
  I think they are just getting better at the edges, MCP/Tool Calls, structured output. This definitely isn't increased intelligence, but it an increase in the value add, not sure the value added equates to training costs or company valuations though. In all reality, I have zero clue how any of these companies remain sustainable. I've tried to host some inference on cloud GPUs and its seems like it would be extremely cost prohibitive with any sort of free plan.
  • layoric 1y ago
    > how any of these companies remain sustainable They don't, they have a big bag of money they are burning through, and working to raise more. Anthropic is in a better position cause they don't have the majority of the public using their free-tier. But, AFAICT, none of the big players are profitable, some might get there, but likely through verticals rather than just model access.
    • tymscar 1y ago
      Doesn’t this mean that realistically even if “the bubble never pops”, at some point money will run dry? Or do these people just bet on the post money world of AI?
      • Aeolun 1y ago
        The money won’t run dry. They’ll just stop providing a free plan when the marginal benefits of having one don’t outweigh the costs any more.
        • fy20 1y ago
          In two years time you'll need to add an 10% Environmental Tax, 25% Displaced Workers Tax, and 50% tip to your OpenAI bills.
          • FridgeSeal 1y ago
            Or at that point, maybe stop using it and just let them go broke?
• NitpickLawyer 1y ago
  > and that LLMs have basically reached a plateau This is the new stochastic parrots meme. Just a few hours ago there was a story on the front page where an LLM based "agent" was given 3 tools to search e-mails and the simple task "find my brother's kid's name", and it was able to systematically work the problem, search, refine the search, and infer the correct name from an e-mail not mentioning anything other than "X's favourite foods" with a link to a youtube video. Come on! That's not to mention things like alphaevolve, microsoft's agentic test demo w/ copilot running a browser, exploring functionality and writing playright tests, and all the advances in coding.
  • hsn915 1y ago
    Is this something that the models from 4 months ago were not able to do?
    • vessenes 1y ago
      For a fair definition of able, yes. Those models had no ability to engage in a search of emails. What’s special about it is that it required no handholding; that is new.
      • camdenreslink 1y ago
        Is this because the models improved, or the tooling around models improved (both visible and not visible to the end user). My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.
        • vessenes 1y ago
          Resist getting your news from Brooklyn journalists. :) Tooling has improved, and the models have. The combo is pretty powerful. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ will give you a flavor of the last six months of improvements. Francois Cholet (ARC AGI: https://arcprize.org/leaderboard https://arcprize.org/leaderboard) has gone from “No current architecture will ever beat ARC” to “o3 has beaten ARC and now we have designed ARC 2”. At the same time, we have the first really useful 1mm token context model available with reasonably good skills across the context window (Gemini Pro 2.5), and that opens up a different category of work altogether. Reasoning models got launched to the world in the last six months, another significant dimension of improvement. TLDR: Massive, massive increase in quality for coding models. And o3 is to my mind over the line people had in mind for generally intelligent in, say, 2018 — o3 alone is a huge improvement launched in the last six months. You can now tell o3 something like: “research the X library and architect a custom extension to that library that interfaces with my weird garage door opener; after writing the architecture implement the extension in (node/python/go) and come back in 20 minutes with something that almost certainly compiles and likely largely interfaces properly, leaving touch-up work to be done.
• strangescript 1y ago
  I have used claude code a ton and I agree, I haven't noticed a single difference since updating. Its summaries I guess a little cleaner, but its has not surprised me at all in ability. I find I am correcting it and re-prompting it as much as I didn't with 3.7 on a typescript codebase. In fact I was kind of shocked how badly it did in a situation where it was editing the wrong file and it never thought to check that more specifically until I forced it to delete all the code and show that nothing changed with regards to what we were looking at.
  • hsn915 1y ago
    I'd go so far as to say Sonnet 3.5 was better than 3.7 At least I personally liked it better.
    • vessenes 1y ago
      I also liked it better but the aider leaderboards are clear that 3.7 was better. I found it extremely over eager as a coding agent but my guess is that it needed different prompting than 3.6
• voiper1 1y ago
  The benchmarks in many ways seem to be very similar to claude 3.7 for most cases. That's nowhere near enough reason to think we've hit a plateau - the pace has been super fast, give it a few more months to call that...! I think the opposite about the features - they aren't gimmicks at all, but indeed they aren't part of the core AI. Rather it's important "tooling" that adjacent to the AI that we need to actually leverage it. The LLM field in popular usage is still in it's infancy. If the models don't improve (but I expect they will), we have a TON of room with these features and how we interact, feed them information, tool calls, etc to greatly improve usability and capability.
• make3 1y ago
  the increases are not as fast, but they're still there. the models are already exceptionally strong, I'm not sure that basic questions can capture differences very well
  • hsn915 1y ago
    Hence, "plateau"
    • j_maffe 1y ago
      "plateau" in the sense that your tests are not capturing the improvements. If your usage isn't using its new capabilities then for you then effectively nothing changed, yes.
    • make3 1y ago
      plateau means stopped
      • camdenreslink 1y ago
        It could mean improving more and more slowly all the time, approaching an asymptote.
    • rxtexit 1y ago
      "I don't have anything to ask the model, so the model hasn't improved" Brilliant! I am pretty much ready to be done talking to human idiots on the internet. It is just so boring after talking to these models.
      • neal_ 1y ago
        :p
• sanex 1y ago
  Well to be fair it's only .3 difference.
• jug 1y ago
  This is my feeling too, across the board. Nowadays, benchmark wins seem to come from tuning, but then causing losses in other areas. o3, o4-mini also hallucinates more than o1 in SimpleQA, PersonQA. Synthetic data seems to cause higher hallucination rates. Reasoning models at even higher risk due to hallucinations risking to throw the model off track at each reasoning step. LLM’s in a generic use sense are done since already earlier this year. OpenAI discovered this when they had to cancel GPT-5 and later released the ”too costly for gains” GPT-4.5 that will be sunset soon. I’m not sure the stock market has factored all this in yet. There needs to be a breakthrough to get us past this place.
• fintechie 1y ago
  It's not that it isn't better, it's actually worse. Seems like the big guys are stuck on a race to overfit for benchmarks, and this is becoming very noticeable.
• _el1s7 1y ago
  Yes. They just need to put out a simple changelog for these model updates, no need to make a big announcement everytime to make it look like it's a whole new thing. And the version numbers are even worse.
• pantsforbirds 1y ago
  It seems MUCH better at tool usage. Just had an example where I asked Sonnet 4 to split a PR I had after we had to revert an upstream commit. I didn't want to lose the work I had done, and I knew it would be a pain to do it manually with git. The model did a fantastic job of iterating through the git commits and deciding what to put into each branch. It got everything right except for a single test that I was able to easily move to the correct branch myself.