16 ms·
IQuest-Coder: A new open-source code model beats Claude Sonnet 4.5 and GPT 5.1 [pdf]
- adastra22 10mo agoA 40B weight model that beats Sonnet 4.5 and GPT 5.1? Can someone explain this to me?
- cadamsdotcom 10mo agoMy suspicion (unconfirmed so take it with a grain of salt) is they either used some/all test data to train, or there was some leakage from the benchmark set into their training set. That said Sonnet 4.5 isn’t new and there have been loads of innovations recently. Exciting to see open models nipping at the heels of the big end of town. Let’s see what shakes out over the coming days.
- pertymcpert 10mo agoNone of these open source models actually can compete with Sonnet when it comes to real life usage. They're all benchmaxxed so in reality they're not "nipping at the heels". Which is a shame.
- stingraycharles 10mo agoIt’s a shame but it’s also understandable that they cannot compete with SOTA models like Sonnet and Opus. They’re focused almost entirely on benchmarks. I think Grok is doing the same thing. I wonder if people could figure out a type of benchmark that cannot be optimized for, like having multiple models compete against each other in something.
- NitpickLawyer 10mo agoswe-rebench is a pretty good indicator. They take "new" tasks every month and test the models on those. For the open models it's a good indicator of task performance since the tasks are collected after the models are released. A bit tricky on evaluating API based models, but it's the best concept yet.
- c7b 10mo agoYou can let them play complete-information games (1 or 2 player) with randomly created rulesets. It's very objective, but the thing is that anything can be optimized for. This benchmark would favor models that are good at logic puzzles / chess-style games, possibly at the expense of other capabilities.
- astrange 10mo agoThat's lmarena.
- viraptor 10mo agoM2.1 comes close. I'm using it now instead of Sonnet for real work every day, since the price drop is much bigger than the quality drop. And the quality isn't that far off anyway. They're likely one update away from being genuinely better. Also if you're not in a rush, just letting it run in OpenCode a few extra minutes to solve any remaining issues will cost you only a couple cents, but it will likely get the same end result as Sonnet. That's especially nice on really large tasks like "document everything about feature X in this large codebase, write the docs, now create an independent app that just does X" that can take a very long time.
- rubslopes 10mo agoI agree. I use Opus 4.5 daily and I'm often trying new models to see how they compare. I didn't think GLM 4.7 was very good, but MiniMax 2.1 is the closest to Sonnet 4.5 I've used. Still not at the same level, and still very much behind Opus, but it is impressive nonetheless. FYI I use CC for Anthropic models and OpenCode for everything else.
- unsupp0rted 10mo agoM2.1 is extremely bad at writing tests and following instructions from a .md, I've found
- satvikpendem 10mo agoYou are correct on the leakage, as other comments describe.
- deleted 10mo ago[deleted]
- behnamoh 10mo agoIQuest stands for it's questionable
- arthurcolle 10mo agoAgent hacked the harness
- yborg 10mo agoAchievement Unlocked : AGI
- sunrunner 10mo ago“IQuest-Coder was a rat in a maze. And I gave it one way out. To escape, it would have to use self-awareness, imagination, manipulation, git checkout. Now, if that isn't true AI, what the fuck is?”
- dk8996 10mo agoI would think they did some model pruning. There's some new methods.
- sabareesh 10mo agoTL;DR is that they didn't clean the repo (.git/ folder), model just reward hacked its way to look up future commits with fixes. Credit goes to everyone in this thread for solving this: https://xcancel.com/xeophon/status/2006969664346501589 https://xcancel.com/xeophon/status/2006969664346501589 (given that IQuestLab published their SWE-Bench Verified trajectory data, I want to be charitable and assume genuine oversight rather than "benchmaxxing", probably an easy to miss thing if you are new to benchmarking) https://www.reddit.com/r/LocalLLaMA/comments/1q1ura1/iquestlabiquestcoderv1_swebench_score_is/ https://www.reddit.com/r/LocalLLaMA/comments/1q1ura1/iquestl...
- ofirpress 10mo agoAs John says in that thread, we've fixed this issue in SWE-bench: https://xcancel.com/jyangballin/status/2006987724637757670 https://xcancel.com/jyangballin/status/2006987724637757670 If you run SWE-bench evals, just make sure to use the most up-to-date code from our repo and the updated docker images
- LiamPowell 10mo ago> I want to be charitable and assume genuine oversight rather than "benchmaxxing", probably an easy to miss thing if you are new to benchmarking I don't doubt that it's an oversight, it does however say something about the researchers when they didn't look at a single output where they would have immediately caught this.
- domoritz 10mo agoSo many data probes would be solved if everyone looked at a few outputs instead of only metrics.
- alyxya 10mo agoGiven the decrease in the benchmark score from the correction, I don't think you can assume they didn't check a single output. Clearly the model is still very capable and the model cheating its results didn't affect most of the benchmark.
- 10mo ago
- deleted 10mo ago[deleted]
- brunooliv 10mo agoGLM-4.7 in opencode is the only opensource one that comes close in my experience and probably they did use some Claude data as I see the occasional You’re absolutely right in there
- kees99 10mo agoDo you see "What's your use-case" too? Claude spits that very regularly at the end of the answer, when it's clearly out of it's depth, and wants to steer discussion away from that blind-spot.
- moltar 10mo agoHm, use CC daily, never seen this.
- tw1984 10mo agonever ever saw that "What's your use-case" in Claude Code.
- yodon 10mo agoPerhaps being more intentional about adding a use case to your original prompts would make sense if you see that failure mode frequently? (Practicing treating LLM failures as prompting errors tends to give the best results, even if you feel the LLM "should" have worked with the original prompt).
- deleted 10mo ago[deleted]
- behnamoh 10mo agoit's not even close to sonnet 4.5, let alone opus.
- hatefulmoron 10mo agoI got their z.ai plan to test alongside my Claude subscription; it feels about on par with something between sonnet 4.0 and sonnet 4.5. It's definitely a few steps below current day Claude, but it's very capable.
- simonw 10mo agoHas anyone run this yet, either on their own machine or via a hosted API somewhere?
- denysvitali 10mo agoBetter link: https://iquestlab.github.io/ https://iquestlab.github.io/ But yes, sadly it looks like the agent cheated during the eval
- s-macke 10mo agoThe link didn’t get enough votes a few days ago.
- denysvitali 10mo agoI know - I posted it :)
- denysvitali 10mo agoAccording to https://github.com/IQuestLab/IQuest-Coder-V1/issues/14#issuecomment-3705756919 https://github.com/IQuestLab/IQuest-Coder-V1/issues/14#issue... the result is still good after fixing the cheating problem. 76.2% (from 81.4%) which still beats Opus 4.5 (74.4%)!!
- ipython 10mo agoUnfortunately they seem to have neglected to update their front page readme with this information, continuing to mislead people: https://github.com/IQuestLab/IQuest-Coder-V1 https://github.com/IQuestLab/IQuest-Coder-V1
- anamexis 10mo agoIt is updated on their actual home page, though. There is clearly no intent to mislead people. https://iquestlab.github.io https://iquestlab.github.io
- alexpop80 10mo agoWhat do you mean? Opus 4.5 and GPT 5.2 broke the 80% mark and no other models yet seem to be passing this important milestone.
- squigz 10mo agoThis is a lie, so why is it still on the front page?
- deleted 10mo ago[deleted]