2 ms·

I don't care about benchmarks. Benchmarks show that Opus 5 is a stronger model than Fable 5 which is obviously not the case. But I do care about capability and

• zorked 1mo ago
  People have been using LLMs for two year. It's not just this week's LLM release that is capable something.
  • blfr 1mo ago
    It's not this week's change. Fable was the step change for programming. And most of truly useful and powerful capabilities arrived in the last eight months.
    • slopinthebag 1mo ago
      People said this for Opus 4.6 too. Every release the models get RLHF'ed into accomplishing a new task and the people who need to do this task think there was a step change.
      • ifwinterco 1mo ago
        To be fair Opus 4.6 was genuinely a really good model when it came out, in fact I'm not sure Opus 5 is even any better. It's definitely a lot slower, though
        • slopinthebag 1mo ago
          Yeah that's kinda my point. I'm not sure if the models have gotten that much smarter, but they're certainly getting more capable. That's not the same thing though. There are things GPT 6.0 can accomplish for me that 5.3 was not able to. But there are also things it still fails at, and it doesn't seem to be much better at the big picture. It does spam about 100x more tests though and I wonder if just RLHFing it to test everything constantly is carrying it more. 6.0 writes so many tests and spends so much time verifying it's work in python sandboxes. Slow as hell but it tends to get things right the first time more which is good, I guess. I don't love the thought of a 500loc feature adding +4000loc due to tests though.
          • ifwinterco 1mo ago
            Yep same with latest Claude models - code isn't really any better than Opus 4.5/4.6, but use 5x as many tokens doing random stuff that's mostly unnecessary. And yeah still for some reason they often can't understand how to set up any project locally without handholding, which is something you'd think an LLM would actually be good at
    • Lapel2742 1mo ago
      > Fable was the step change for programming. AFAIK: Mistral does not even try to compete in this field. There are other use cases for LLMs beside coding. As Mistral AI wrote: > During the first wave of generative AI, the central question was who could build the most powerful model. Organizations and governments are now asking a different one: how to harness the power of AI for their mission-critical needs without surrendering control over the infrastructure and intelligence loop. Demand for that combination of performance with control, choice and independence is growing internationally, as enterprises and governments weigh the long-term technology dependencies, data governance requirements and deployment choices that come with any AI investment. > Mistral is the only AI company in the world building the full stack required to answer that question: open-weight models, the infrastructure and the compute capacity they run on, and the products that bring them into production; ensuring that customers are never locked into a single vendor's roadmap, pricing or availability. > Mistral’s full-stack and open approach also allows organizations to build on it without exposing their most valuable data, workflows and institutional knowledge to anyone outside their own walls. That's what makes Mistral’s stack the sovereign AI layer, meaning retaining control across four dimensions: data that stays inside the organization's boundaries, models that are controllable and customizable, compute that is private and predictable, and systems in production that are fully controllable and auditable.
      • nicce 1mo ago
        >> Fable was the step change for programming > AFAIK: Mistral does not even try to compete in this field. They have released models speficially for programming that ”vibe coding” would be safer. https://mistral.ai/news/leanstral/ https://mistral.ai/news/leanstral/
    • raverbashing 1mo ago
      If the choice is between Mistral and no AI, I'll take Mistral any day Even those old llama models were ok for coding Yes yes they won't be like Claude's fire and forget (until you see how many tokens you burned to write "Hello World")
  • piyush_patel_ha 1mo ago
    A capability isn't binary. There is a massive difference between can produce an impressive demo and can reliably complete the task without a human babysitting it.
    • antupis 1mo ago
      Yup, new SOTA models especially with high/xhigh/max reasoning too often overengineer solutions, good for benchmarks that usually measure task completion, bad for normal development where you don't want 'rewrite in rust and 1k LOC unit tests' style solutions when agent does mundane bug fixes.
      • nananana9 1mo ago
        When it comes to mundane bug fixes the value is in actually finding the cause of the bug, and I find SOTA models way outperform smaller ones here. I don't care about their output - I can write the correct 5 line patch myself once I understand what's wrong.
  • guywithahat 29d ago
    People fawn over AI brands now like cars and it's silly. OpenAI and Anthropic have been flipping spots for best LLM coder for the last two years and to say one is better feels silly; I've been using them both and they're very similar with different personalities. Recently Grok has become competitive in many aspects, and while I don't have much experience with Gemini it seems to come and go in terms of coding quality. Saying only anthropic models are competitive frontier coding models is out of touch with the space imo
• mdp2021 1mo ago
  > don't care about benchmarks You must care about good benchmarks (identify those that have relevance).
  • tripledry 1mo ago
    Genuinely interested, which ones do you think have relevance? If I read forums and talk to people IRL most have differing opinions what model is best. Yes, for me it's pretty clear Opus is better than earlier models, but it's at least not obvious to me that the later are significant improvements.
    • mdp2021 28d ago
      Sorry for the delay. It can be subjective, at this stage of product availability. Personally, I hate to be frustrated by gross intellectual faults, so I did some research in the past about the best benchmarks to assess pure (simulated, apparent) intelligence. (The quality of the found benchmarks may not reflect what the models seem to do in practice, so one's experience should be compared to the raw numbers out of the benchmarks.) Good ideas emerge in the field: it was proposed and discussed on these very pages that the LLM should be able to solve "murder mysteries", for example (alongside the pattern recognition problems in which IQ tests consist, etc.). Moreover, the LLM shall not delirate. It is an intrinsic issue with the current architectures (they do not mirror the "Foundational theory of Knowledge", which requires confidence values and relations of foundation between notions), but it is a problem with more or less presence per model. Artificial Analysis has introduced a metric for that. Moreover again, I want an output style that works well for the purpose - must not be a clashing style like "youngspeak" ("like, awsome") or "paternalistspeak" ("when a planet likes another very much they are attracted...") or "sycophantspeak" ("your question is so deep and interesting") or "wetspeak" ("you can do it, feel this not that")... So, for example, I very much preferred Kimi k2 to gpt-oss-120b. I doubt there are benchmarks for this - "seriousspeak", "maturespeak" - but there should be.