6 ms·
It seems to really be a nice step-up and is getting quite close to the frontier. I wish they'd start focusing on the reasoning efficiency now, though. I have a
by Tiberium 4mo ago
It seems to really be a nice step-up and is getting quite close to the frontier. I wish they'd start focusing on the reasoning efficiency now, though. I have a simple (relatively) test task to evaluate LLMs: writing a simple math evaluator library in Nim (it's about 400-600 lines total max), and GLM 5.2 (xhigh which maps to max effort) spent over 15 minutes (!) reasoning, spending about 45k tokens, before it finally wrote the first file.
I know it's hard to improve on that, but now that their models are good enough at raw intelligence, I think this should become a higher priority task.
Currently on https://artificialanalysis.ai/#output-tokens https://artificialanalysis.ai/#output-tokens GPT 5.5 xhigh spends 16k tokens total on average, GPT 5.5 high is 10k, Fable 5 33k, Opus 4.8 41k, GLM 5.2 is 42k. GPT 5.5 is extremely reasoning efficient.
Of course if you convert those values to actual request cost, GLM 5.2 will probably beat GPT 5.5/Opus 4.8, but speed matters for a lot of people, I think.
- bertili 4mo agoThis is GLM 5.2 Max. GLM 5.2 High which use less than half[1] the tokens. [1] https://z.ai/blog/glm-5.2 https://z.ai/blog/glm-5.2
- Tiberium 4mo agoYes, but the Artificial Analysis result is also from GLM 5.2 (max), not high.
- andai 4mo agoThey have this with a lot of models, measuring only the max setting, while the one you'd actually want to use for most tasks is much lower.
- epolanski 4mo agoFor the brief period with had Fable, I never had to use it above medium. Low nailed the overwhelming majority of mundane tasks on it's own, medium was good for more complex stuff.
- vorticalbox 4mo agoThis is a problem I find with opus is will spend so long thinking then going “but wait what if” To point where I stop it and simple tell it to “start writing code you can work it out as you go along” Seems writers block also effects LLM
- epolanski 4mo agoFable was 20 times worse on that. It's clear it was the vibe coding model, as like no other model before, fully turned you into his assistant instead of the other way around.
- deleted 4mo ago[deleted]
- RyanHamilton 4mo agoCould it be possible, these firms are optimizing for two things: a) Better performance. b) Gathering data from you to further improve performance later. I've also found the huge amount of planning rather than iteration frustrating. I've felt like I'm teaching a junior!
- epolanski 4mo agoI think they simply optimize around E2E benchmarks, none of those benchmarks is designed as multi turn assistance to the user, but going from a prompt straight to the final solution.
- celrod 4mo agoExactly. How can "we" develop and encourage benchmarks for multi-turn user assistance? That is what I want. I feel like the models and harnesses push much too hard against this workflow -- that they push you towards letting go and vibe coding, with only your discipline (and desire for a quality and maintainable product) holding it back.
- happyPersonR 4mo ago
- deleted 4mo ago[deleted]
- benjiro29 4mo agoGLM 5.2 Max = Opus 4.8 Max in thinking behavior. The thinking chain is so similar, and so is the amount of token usage on the output. If you want reasonable token usage, you need to run it GLM 5.2 at High. There is little drop in quality from Max to High (for most tasks). And it cuts token usage by 2 a 2.5x. GLM 5.2, Max is really something you only need for complex tasks. In essence, GLM 5.2 is Opus 4.8 its little brother, at a way, WAY cheaper price. There has been really no training on Opus models going on, really, none i tell you! /sarcasm
- vitalyan123 4mo agodistillation of thinking models is not particularly effective - both "Open"AI and Misanthropic don't show you the real chain of thought, only its severely downscaled version. both do everything in their power to combat such outrageous copyright infringement, so the bulk of unethically scrapped data the Chinese have is from several generations ago.
- duskdozer 4mo ago>such outrageous copyright infringement Sarcasm, considering the source of their own training data?
- orphea 4mo agoNarrator: it was sarcasm, indeed.
- baron3dl 4mo agoIP for me, not thee.
- margalabargala 4mo agoConsidering they called the company "Misanthropic", sarcasm is a safe bet.
- duskdozer 4mo ago
- cmrdporcupine 4mo ago> Of course if you convert those values to actual request cost, GLM 5.2 will probably beat GPT 5.5/Opus 4.8, but speed matters for a lot of people, I think. GLM5.2 ends up being far more expensive than I thought it would be when I tried it on openrouter. I ground through $5 USD worth of tokens quite quickly. And this was high, not max.
- robmccoll 4mo agoThat's interesting. I gave nearly the same task to Gemma4 31b as a test yesterday. Write a symbolic math engine in Typescript that can perform evaluation and simple expression reductions over +-/*(). It performed the task correctly with minimal reasoning - much fewer reasoning tokens than output tokens.
- gbingles 4mo agoTbh, so what? I googled "symbolic math engine in Typescript that can perform evaluation and simple expression reductions over +-/*()" and got what looks to be viable answers without using any AI model at all. Reciting well established things from memory isn't terribly interesting. Show it a novel codebase and have it implement something within it.
- drob518 4mo agoSo, a natural question would be why a model would ever get it wrong?
- SubiculumCode 4mo agoTBH, while your point is a fair one, your attitude is off-putting and needlessly condescending.
- h14h 4mo agoHopefully the recent work Moonshot did with Kimi K2.7 Code trickles in to the other open-model labs. Per AA, while K2.7 Code is roughly on par w/ K2.6 in terms of intelligence, it uses half the output tokens to get there.
- h14h 4mo agoI've been doing some testing with GLM 5.2 on Fireworks and it looks like the "High" reasoning level uses fewer tokens than even K2.7 Code by a considerable margin (roughly half). Don't have any evals indicating how it compares on upper-bound quality, but for a well-defined task it seems like GLM 5.2 on "High" is remarkably token efficient. Looking forward to seeing where it lands on the AA index.
- rdsubhas 4mo agoAs per stats in other comments, it is frontier, not close to frontier.
- esafak 4mo agoI agree. I've noticed that it is quite smart but it has a tendency to doubt itself and overthink. I monitor its internal dialogue and prod it when it does this. They need to optimize the chain of thought early stopping.
- deleted 4mo ago[deleted]
- alexjplant 4mo ago> It seems to really be a nice step-up and is getting quite close to the frontier. IMHO it's already surpassed them. I vastly prefer my personal GLM and OpenCode setup to the Claude Code and Opus one that I have to use at work. The former makes way fewer StackOverflow brogrammer-tier mistakes and is considerably better at following instructions. The harness UX is also vastly superior as it doesn't ignore, randomly change, or incorrectly report settings. Maybe it's the harness and I'd have even greater success with OpenCode and Anthropic, but I think it safe to say that Anthropic's moat is evaporating.
- carter2099 4mo agoYou would be surprised at how much of an impact the harness has. I switched to Pi and chinese open source models, and models that _I know_ are less capable than sonnet outperform my sonnet + claude code stack at work.
- nurumaik 4mo agoYou asked for maximum effort, you got maximum effort
- xyzsparetimexyz 4mo agoReminiscent of https://en.wikipedia.org/wiki/Portia_(spider) https://en.wikipedia.org/wiki/Portia_(spider)
- HWR_14 4mo agoI thought you could not compare tokens across models because their cost and speed was so different between models.
- gateonai 4mo ago[flagged]
- abgruszecki 4mo agoAgreed that models should get better at working with rare programming languages like Nim! Using them tends to confuse agents a lot in general. We're working on a paper right now where we compare how token-efficient models are when trying to implement the exact same program in different programming languages, and that's one of the trends we're seeing.