4 ms·

IMHO there is a point where incremental model quality will hit diminishing returns. It is like comparing an 8K display to a 16K display because at normal viewi

• snek_case 6mo ago
  It probably depends what you're using the models for. If you use them for web search, summarizing web pages, I can imagine there's a plateau and we're probably already hitting it. For coding though, there is kind of no limit to the complexity of software. The more invariants and potential interactions the model can be aware of, the better presumably. It can handle larger codebases. Probably past the point where humans could work on said codebases unassisted (which brings other potential problems).
  • Bolwin 6mo ago
    > summarizing web pages For summarizing creative writing, I've found Opus and Gemini 3 pro are still only okay and actively bad once it gets over 15K tokens or so. A lot of long context and attention improvements have been focused on Needle in a Haystack type scenarios, which is the opposite of what summarization needs.
• aray07 6mo ago
  yeah thats is my biggest issue - im okay with paying 20-30% more but what is the ROI? i dont see an equivalent improvement in performance. Anthropic hasnt published any data around what these improvements are - just some vague “better instruction following"
  • margorczynski 6mo ago
    The other thing is most people don't really care about price per token or whatever but how much it will cost to execute (successfully) a task they want. It doesn't matter if a model is e.g. 30% cheaper to use than another (token-wise) but I need to burn 2x more tokens to get the same acceptable result.
  • Bridged7756 6mo ago
    Its enshittificating real fast. They'll just keep releasing model after model, more expensive than the last, marginal gains, but touted as "the next thing". Evangelists will say that they're afraid, it's the future, in 6 months it's all over. Anthropic will keep astroturfing on Reddit. CEOs will make even more outlandish claims. You raised a good point, what's a good metric for LLM performance? There's surely all the benchmarks out there, but aren't they one and done? Usually at release? What keeps checking the performance of those models. At this point it's just by feel. People say models have been dumbed down, and that's it. I think the actual future is open source models. Problem is, they don't have the huge marketing budget Anthropic or OpenAI does.
    • conductr 6mo ago
      This is most likely trajectory I fear. It reminds me a lot of Oracle, where they rebrand and reskin products just to change pricing/marketing without adding anything.
      • skydhash 6mo ago
        Win 10, win 11, all the recent macOS,… could have been released as features and not new products
        • conductr 6mo ago
          My only problem lumping them in is a huge part of those products are the UI and UX. It’s what you interact with as a user. So, a reskin with no new features is a valid version in my opinion. It’s not always for the best mind you, but I do think major changes to the GUI or UX do deserve a version indicator given they are relatively infrequent. Without some requirement of course. Nobody buys Oracle because of the UI or upgrades because of a new UI. It’s the underlying tool that matters and if my ugly version is working there is no incentive to upgrade to a prettier version.
• ZeroCool2u 6mo ago
  Whenever we get the locally runnable 4k models things are going to get really awkward for the big 3 labs. Well at least Google will still have their ad revenue I guess.
  • robot_jesus 6mo ago
    They're not perfect but the local model game is progressing so quickly that they're impossible to ignore. I've only played around with the new qwen 3.6 models for a few minutes (it's damn impressive) but this weekend's project is to really put it through its paces. If I can get the performance I'm seeing out of free models on a 6-year-old Macbook Pro M1, it's a sign of things to come. Frontier models will have their place for 1) extensive integrations and tooling and 2) massive context windows. But I could see a very real local-first near future where a good portion of compute and inference is run locally and only goes to a frontier model as needed.
    • UncleOxidant 6mo ago
      I've had really good results form qwen3-coder-next. I'm hoping we get a qwen3.6-coder soon since claude seems to get less-and-less available on the pro plan.
  • UncleOxidant 6mo ago
    Given how little claude usage they've been giving us on the "pro" plan lately, I've started doing more with the various open Qwen3.* models. Both Qwen3-coder-next and Qwen3.5-27b have been giving me good results and their 3.6 models are starting to be released. I think Anthropic may be shooting themselves in the foot here as more people start moving to local models due to costs and/or availability. Are the Qwen models as good as Claude right now? No. But they're getting close to as good as Claude sonnet was 9 months to a year ago (prior to 4.5, around 4.0). If I need some complex planning I save that for claude and have the Qwen models do the implementation.
    • blurbleblurble 6mo ago
      I was thinking the exact same thing just now as I load up qwen3.6 into hermes agent and all while fantasizing that it will replace opus 4.7. It might not actually but seems like we're on the verge of that. Lately I've been wondering too just how large these proprietary "ultra powerful frontier models" really are. It wouldn't shock me if the default models aren't actually just some kind of crazy MoE thing with only a very small number of active params but a huge pool of experts to draw from for world knowledge.
• mlinsey 6mo ago
  I agree, but also the model intelligence is quite spikey. There are areas of intelligence that I don't care at all about, except as proxies for general improvement (this includes knowledge based benchmarks like Humanity's Last Exam, as well as proving math theorems etc). There are other areas of intelligence where I would gladly pay more, even 10X more, if it meant meaningful improvements: tool use, instruction following, judgement/"common sense", learning from experience, taste, etc. Some of these are seeing some progress, others seem inherent to the current LLM+chain of thought reasoning paradigm.
  • manmal 6mo ago
    Common sense isn’t a language pattern. I doubt this will ever work w/ LLMs.
    • mlinsey 6mo ago
      The models that we are paying to generate tokens are already not really just LLMs, as anyone studying language models ten years ago (or someone who describes them as "next token predictors") would understand them. Doing a bunch of reinforcement learning so that a model performs better at ssh'ing into my server and debugging my app is already realllly stretching the definition of "language pattern". I think when we do get AI that can perform as well as a human at functionally all tasks, they will be multi-paradigm systems; some components will not resemble anything in any commercial system today, but one component will be recognizably LLM-like, and act as an essential communication layer.
• nisegami 6mo ago
  >IMHO there is a point where incremental model quality will hit diminishing returns. It's not necessary a single discrete point I think. In my experience, it's tied to the quality/power of your harness and tooling. More powerful tooling has made revealed differences between models that were previously not easy to notice. This matches your display analogy, because I'm essentially saying that the point at which display resolution improvements are imperceptible matters on how far you sit.
• simplyluke 6mo ago
  I'm seeing a lot of sentiment, and agree with a lot of it, that opus 4.6 un-nerfed is there already and for many if not most software use cases there's more value to be had in tooling, speed, and cost than raw model intelligence.
• _pdp_ 6mo ago
  Longer version of the comment https://www.linkedin.com/pulse/imperceptible-upgrade-petko-d-petkov-o2yxe https://www.linkedin.com/pulse/imperceptible-upgrade-petko-d...
• deleted 6mo ago
  [deleted]
• levocardia 6mo ago
  Depends a lot on the task demands. "Got 95% of the way to designing a successful drug" and "Got 100% of the way" is a huge difference in terms of value, and that small bump in intelligence would justify a few orders of magnitude more in cost.
  • 9dev 6mo ago
    But that objective measure is exactly what we’re lacking in programming: There is often many ways to skin a cat, but the model only takes one. Without knowing about those it didn’t take, how do you judge the quality of a new model?
    • altern8 6mo ago
      I would say following instructions. If Claude understood what you mean better without you having to over explain it would be an improvement
      • snoman 6mo ago
        I agree with you, but my gut tells me that a lot of people don’t know what a good outcome should/could look like and are accepting whatever it delivers.
• iLoveOncall 6mo ago
  > IMHO there is a point where incremental model quality will hit diminishing returns. You mean a couple of years ago?
• wellthisisgreat 6mo ago
  Does anyone here use 8k display for work? Does it make sense over 4k? I was always wondering where that breaking point for cost/peformance is for displays. I use 4K 27” and it’s noticeably much better for text than 1440p@27 but no idea if the next/ and final stop is 6k or 8k?
  • zozbot234 6mo ago
    Even 4k turns out to be overkill if you're looking at the whole screen and a pixel-perfect display. By human visual acuity, 1440p ought to be enough, and even that's taking a safety margin over 1080p to account for the crispness of typical text.
    • solenoid0937 6mo ago
      1440p is enough if you haven't experienced anything else. Even the jump from 4k to 5-6k is quite noticeable on a 27" monitor. I switched to the Studio Display XDR and it is noticeably better than my 4k displays and my 1440p displays feel positively ancient and near unusable for text.
      • zozbot234 6mo ago
        That's great for contrast, color fidelity and compatibility with the Apple Mac. But the resolution is quite overkill.
• mgraczyk 6mo ago
  This will probably happen but I wouldn't plan on it happening soon
• highfrequency 6mo ago
  I believe that's why 90% of the focus in these firms is on coding. There is a natural difficulty ramp-up that doesn't end anytime soon: you could imagine LLMs creating a line of code, a function, a file, a library, a codebase. The problem gets harder and harder and is still economically relevant very high into the difficulty ladder. Unlike basic natural language queries which saturate difficulty early. This is also why I don't see the models getting commoditized anytime soon - the dimensionality of LLM output that is economically relevant keeps growing linearly for coding (therefore the possibility space of LLM outputs grows exponentially) which keeps the frontier nontrivial and thus not commoditized. In contrast, there is not much demand for 100 page articles written by LLMs in response to basic conversational questions, therefore the models are basically commoditized at answering conversational questions because they have already saturated the difficulty/usefulness curve.
  • Aperocky 6mo ago
    > the dimensionality of LLM output that is economically relevant keeps growing linearly for coding Doubt. Yes. there was at one point it suddenly became useful to write code in a general sense. I have seen almost no improvement in department of architecting, operations and gaslighting. In fact gaslighting has gotten worse. Entire output based on wrong assumption that it hid, almost intentionally. And I had to create very dedicated, non-agentic tools to combat this. And all of this with latest Opus line.
    • skydhash 6mo ago
      Also doubt. But most likely because of organizational inertia. After a while, you’re mostly focused on small problems and big features are rare. You solution is quasi done. But now each new change is harder because you don’t want to broke assumptions that have become hard requirements.
    • zarzavat 6mo ago
      Agreed. The proprietary nature of these tools is a huge impediment to their usefulness. A intelligence plateau will happen sooner or later (my bet is on sooner), and when it does the open models will catch up. And everybody will be using open models and open source agents because they're so much more flexible.
• zadkey 6mo ago
  yeah there needs to be a corresponding increment improvement in model archetecture.
• Rapzid 6mo ago
  At normal viewing distance(let's say cinema FOV) most people won't see a difference between 4k and 8k never mind 16k. And it's not that they "don't notice" it's that they physically can't distinguish finer angular separation.
• jasonjmcghee 6mo ago
  It's more like, if it gets it right 99% of the time, that sounds incredible. Until it's making 100k decisions a day and many are dependent on previous results.
• AlfeG 6mo ago
  At this point, I still don't see a reason to use Opus. I'm happy with Sonnet's performance for a third of the price. Tried several times with not a big gain.
• naasking 6mo ago
  Diminishing returns are inevitable, agreed, but it's not clear we're near that point yet.