4 ms·
Early benchmarks show tremendous improvement over Kimi K2 Thinking, which didn't perform well on our benchmarks (and we do use best available quantization). Ki
by gertlabs 6mo ago
Early benchmarks show tremendous improvement over Kimi K2 Thinking, which didn't perform well on our benchmarks (and we do use best available quantization).
Kimi K2.6 is currently the top open weights model in one-shot coding reasoning, a little better than GLM 5.1, and still a strong contender against SOTA models from ~3 months ago (comparable to Gemini 3.1 Pro Preview).
Agentic tests are still running, check back tomorrow. Open weights models typically struggle with longer contexts in agentic workflows, but GLM 5.1 still handled them very well, so I'm curious how Kimi ends up. Both the old Kimi and the new model are on the slower side, so that's a consideration that makes them probably less usable for agentic coding work, regardless. The old Kimi K2 model was severely benchmaxxed, and was only really interesting in the context of generating more variation and temperature, not for solving hard problems. The new one is a much stronger generalist.
Overall, the field of open weights models is looking fantastic. A new near-frontier release every week, it seems.
Comprehensive, difficult to game benchmarks at https://gertlabs.com/?mode=oneshot_coding https://gertlabs.com/?mode=oneshot_coding
- cmrdporcupine 6mo agoSurprised to see such variance per language
- gertlabs 6mo agoIt's interesting; I can only speculate as to the underlying reason. When given enough time, models outperform in Rust/C++ in longer agentic tasks, and actually perform worst in Python. For tasks that aren't judged on code speed. https://gertlabs.com/?mode=agentic_coding https://gertlabs.com/?mode=agentic_coding
- edude03 6mo agoIt makes sense when you consider LLMs don't generalize very well, so they're heavily dependent on how good (how varied as well as how high quality) the training data is
- cmrdporcupine 6mo agoWell it might explain why pro-Claude vs pro-Codex people keep talking past each other on this forum. I see people all the time assuming that anybody who likes Codex must be some sort of bot because of their own biases, but I work almost exclusively in Rust and find Codex extremely competent (and a much better overall engineer), don't trust Claude/Opus at all... but I see in this bench it scores lower on TypeScript etc. than Opus does.
- tmaly 6mo agoHow would K2.6 compare to Sonnet 4.6 both price and performance wise?
- Mattwmaster58 6mo agoIn terms of raw token cost, I've seen a couple providers at (all prices in terms of Mtok) $0.95 input/$0.15 cache input/$5 output vs $3 input/$15 output for sonnet. Task prices of courses will be more interesting - a dumber model may use more tokens to get to the same goal.
- knollimar 6mo agowait why compare 2.6 to 2 instead of to 2.5?
- gertlabs 6mo agoGood question. We missed that release entirely. Our automated model checker only went live 2 months ago so they were manually curated prior to that. I'm adding it now. It'll be live in ~12 hours.
- gertlabs 6mo agoUpdate: Kimi K2.5 one-shot results are live. It wasn't a noteworthy release compared to K2.6: https://gertlabs.com/?mode=oneshot_coding https://gertlabs.com/?mode=oneshot_coding
- DeathArrow 6mo agoCan you add C# to supported languages? It's widely used and it be helpful for people and companies to see how different models fare against each other.
- gertlabs 6mo agoGood idea.
- esperent 6mo agoI'm looking at your table now - is there a reason why you don't include cost? If Opus 4.7 is the winner but costs e.g. 5x as much, that's important information.
- gertlabs 6mo agoWe recently added cost (last week), so data is sparse. Check back in a few weeks and it will be represented somewhere on the homepage, probably in the Efficiency Chart at the bottom. We also plan to show model performance deviation over time after we collect more data. I'm interested to hear about any other data representations you'd like to see, too. The goal is to convey the most important information as densely as possible, without too much clutter.
- DeathArrow 6mo ago>I'm interested to hear about any other data representations you'd like to see, too It would be nice if you can show how much the models drift from the instructions over time
- gertlabs 6mo agoNot sure what you mean. Time series chart of model performance over time to see if proprietary models get degraded? That's in the works, but we will need a couple months more data collection before launch.
- DeathArrow 6mo agoYes, probably performance helps. The idea is that the larger a coding task is and the longer the coding agent is, the higher the chance is for the agent to not follow the rules and guidelines.
- freely0085 6mo agoCan you add Qwen 3.6 max to the leaderboard?
- gertlabs 6mo agoWe will as soon as API access is widely available. Once a model goes live, we typically have one-shot reasoning benchmarks up in ~8 hours and comprehensive agentic/combined benchmarks up after 24-48 hours. We're working on building relationships with each lab to have the results before launch.
- DustinKlent 6mo agoCool website. I don't understand enough about the various benchmarks or how they're done to judge whether or not anything is accurate, but I love the layout and features especially the spectator feature which is pretty cool. One thing, I saw the "Market simulator" spectator feature but didn't see a corresponding benchmark for that. Is it "Finance" or "Betting" or "Trading"?
- gertlabs 6mo agoThanks -- that one is categorized under Trading/Financial, whereas betting is reserved for games like Pot Limit Omaha Hilo. That's a good idea for a feature request, including the tags for the spectatable demo games.