4 ms·
GLM 5.2 Performance Benchmarks
- DeathArrow 4mo agoOne or two more releases and they will reach Fable level.
- vitalyan123 4mo agoby then there will be Fable 5.21, again 5% ahead of every other SotA while still only 500% the size.
- mjhay 4mo agoThere’s no way Anthropic can keep jacking up the prices like this for every marginally better model. I think even tokenmaxxing companies are going to soon balk at $50/million output tokens.
- theplumber 4mo agoAnthropic wants to ban the alternatives through regulation and ideally provide differential access with differential pricing.
- Tadpole9181 4mo agoAh, yes, Anthropic, the company who just got regulated and has an ongoing open dispute with the government, wants to regulate other companies with their copious political capital.
- deagle50 4mo agocorrect
- lanycrost 4mo agoIt's always nice to see how open source models growing, hope we will have good performance with lower tier hardware some day.
- wongarsu 4mo agoIt does really well on "AA-Omniscience Non-Hallucination Rate", far higher than DeepSeek, GPT 5.5 or Fable. I really like that benchmark because it's one of the few benchmarks that allows LLMs to elect not to answer if they are unsure and punishes them for trying to bullshit their way through the benchmark
- andai 4mo agoThis implies that other benchmarks (for which every AI provider is optimizing?) are actively encouraging bullshitting?
- whimblepop 4mo agoBullshitting is how LLMs work. It doesn't require active encouragement. All it takes is a machine without consciousness or physical access to the world and an actually-lived life. A training set that contains lots of confident answers and few to no refusals doesn't help either.
- otabdeveloper4 4mo agoIt's simpler than that. An LLM outputs tokens, one-by-one. It stops the loop if it outputs the end-of-text token. Which is, of course, statistically much rarer than any other kind of token. (This is why you cannot, in general, prompt an LLM with something like "don't answer if the result is correct". It has to output something, by design.)
- wongarsu 4mo agoYes. Most benchmarks just measure how many answers are correct. The best way to optimize that is to confidently state something, in hopes it's correct. Which is exactly how most LLMs behave, despite plenty of evidence that they do know whether they "know" something
- Imustaskforhelp 4mo agoif this is the case, then GLM 5.2 model seems better than gpt 5.5 or maybe even "Fable" depending upon what you are trying to achieve. Fable model being removed from Anthropic because of security concerns by the US government (or well, also partially because of the personal vendetta between US govt and Anthropic)
- sourcecodeplz 4mo agostill quite verbose at 140m output tokens, but this is on max thinking. high should do better.
- theturtletalks 4mo agoI want to trust their benchmarks but when they have Muse Spark over GPT-5.5, it gives me pause.
- mdasen 4mo agoWhere do you see that? I see they have GPT-5.5 (xhigh) at 55, GPT-5.5 (high) at 53, and Muse Spark at 43. Muse Spark does beat GPT-5.4 mini (xhigh) which scores 40, but the key there is "mini". In the coding index, GPT-5.5 gets 59.1, 58.5, 56.2, and 52.1 for xhigh, high, medium, and low while Muse Spark is behind at 47.5. For agentic, GPT-5.5 gets 74.1, 72.0, 69.4, and 59.7 (xhigh, high, medium, low) while Muse Spark gets 62.0 (beating only GPT-5.5 low). GPT-5.5 only gets beaten by Opus 4.8 in their general index, is the top spot for coding, and is #3 behind Opus 4.8 and GLM-5.2 for agentic (excluding Fable 5 which takes the top spot, but is unavailable).
- theturtletalks 4mo agoI was mistaken, it was Arena that has Muse Spark above GPT-5.5. GLM 5.2 not only tops that but many other benchmarks. Hopefully it is SOTA.
- ChrisArchitect 4mo agoSome more discussion: https://news.ycombinator.com/item?id=48567759 https://news.ycombinator.com/item?id=48567759
- XCSme 4mo agoI also tested it[0]: quite similar to GLM 5, a few percent better, 30% faster and 50% more expensive. [0]: https://aibenchy.com/?q=glm https://aibenchy.com/?q=glm
- XCSme 4mo agoPS: Just added a cool feature, so you can filter the leaderboard for multiple models at once, by using a comma, like: https://aibenchy.com/?q=glm,claude https://aibenchy.com/?q=glm,claude
- lousken 4mo agostill 1/4 of the price of anthropic and openai models though
- benxh 4mo agobenchmark where gemini flash is better than fable btw.
- XCSme 4mo agoWell, most people were not liking Fable when it was available anyway, because it refused to answer questions very often.
- margalabargala 4mo agoAnd therefore it scores worse on benchmarks?
- XCSme 4mo agoOn some it does yes, also in real usage. It avoided answering 2/21 tests in this specific benchmark mark, that's already 90% max score already.
- margalabargala 4mo agoI'm glad those tests apparently work out for you but a benchmark where three of the top 5 models are different flavors of Gemini Flash and zero are anything by Anthropic, is just so wildly divergent from my personal experience with the models that it's not useful to me. Whatever it is you're measuring, it's not anything related to what I use models for.
- hemkeshr 4mo agoLocal models are already useful today. The next milestone is getting this level of performance onto truly affordable hardware.
- SV_BubbleTime 4mo agoNVidia has less than zero reason to ship cards ideal for this at low prices. AMD’s stock price reflects a hope they launch a CUDA alternative. But this is unlikely for the near future. There is a lot of interest in preventing China coming in with cheap AI hardware. So I expect the direction to be good local models that few can run effectively.
- theplumber 4mo agoThe Chinese will flood the market with cheap AI chips just like they did with EV cars. As consumers we can’t thank them enough.
- binary132 4mo agoI think it will eventually result in regulation and a potential grey market, and/or implosion of the centralized LLM services — I doubt they can keep hardware from becoming cheaper forever, and diminishing returns will make consumer hardware suitable for all but the hardest problems. At that point, the hardware “moat” will be completely gone and have become an extreme unrecoverable sunk cost.
- theplumber 4mo agoWell you have tariffs and bans on EVs as well so surely there will be bans and tariffs as well on AI products and chips but for people who really want the chips and models we know they can get it. I expect a market like it used to be for pirated content
- kajman 4mo agoI'm cautiously optimistic that anti-conpetitive action against hardware will fail. There's a lot of money willing to fight for cheaper inference. The same can't be said for providing consumers with cheaper cars. I can't say I'm as optimistic about there continuing to be an open market for foreign LLMs.
- gertlabs 4mo agoOn our multi-agent coding and reasoning evaluations, GLM 5.2 is the first model we've tested that crossed the threshold of being on par with or better than Opus 4.6 (although as usual, we have GLM 5.2 and most other Chinese models a bit below most other benchmarks with test methodologies that are more vulnerable to benchmaxxing). Data at https://gertlabs.com/rankings https://gertlabs.com/rankings
- fcpk 4mo agotangent question: Claude code seems to be very much loved and suggested by most major Chinese LLM using the env vars to change the server. that however means you lose a lot of anthropic tools like auto mode, running shells, monitors/crons. is there a way to get those with non anthropic plans?
- ttoinou 4mo agoWould be interesting to see if we can make this model smaller with REAP + unsloth dynamic quant. It might become 4x cheaper to run for similar quality output
- linzhangrun 4mo agoCoding Plan is completely unavailable