4 ms·
For what? It's really hard to say what model is "generally" better then another, as they're all better/worse at specific things. My own benchmarks has a bunch
by diggan 1y ago
For what? It's really hard to say what model is "generally" better then another, as they're all better/worse at specific things.
My own benchmarks has a bunch of different tasks I use various local models for, and I run it when I wanna see if a new model is better than the existing ones I use. The output is basically a markdown table with a description of which model is best for what task.
They're being sold as general purpose things that are better/worse at everything but reality doesn't reflect this, they all have very specific tasks they're worse/better at, and the only way to find that out is by having a private benchmark you run yourself.
- kelvinjps10 1y agocoding? they are coding models? what specific tasks is one performing better than the other?
- diggan 1y agoThey may be, but there are lots of languages, lots of approaches, lots of methodologies and just a ton of different ways to "code", coding isn't one homogeneous activity that one model beats all the other models at. > what specific tasks is one performing better than the other? That's exactly why you create your own benchmark, so you can figure that out by just having a list of models, instead of testing each individually and basing it on "feels better".
- reverius42 1y ago> coding isn't one homogeneous activity that one model beats all the other models at If you can't even replace one coding model with another, it's hard to imagine you can replace human coders with coding models.
- diggan 1y agoWhat you mean "can't even replace"? You can, nothing in my comment says you cannot?
- Philpax 1y agoYou probably can't replace a seasoned COBOL programmer with a seasoned Haskell programmer. Does that mean that either person is bad at programming as a whole?
- reverius42 1y agoThis was my point -- if programmers are not fungible, how can companies claim to be replacing them by the thousands with AI?
- Philpax 1y agoYou don't need to use the same model/system for every task. "AI" isn't a monolith; there's a spectrum of solutions for a spectrum of problems, and figuring out what's applicable to your problem today is one of the larger problems of deployment.
- whimsicalism 1y agoglm 4.5 is not a coding model
- simonw 1y agoIt may not be code-only, but it was trained extensively for coding: > Our base model undergoes several training stages. During pre-training, the model is first trained on 15T tokens of a general pre-training corpus, followed by 7T tokens of a code & reasoning corpus. After pre-training, we introduce additional stages to further enhance the model's performance on key downstream domains. From my notes here: https://simonwillison.net/2025/Jul/28/glm-45/ https://simonwillison.net/2025/Jul/28/glm-45/
- whimsicalism 1y agoyes, all reasoning models currently are, but it’s not like ds coder or qwen coder
- simonw 1y agoI don't see how the training process for GLM-4.5 is materially different from that used for Qwen3-235B-A22B-Instruct-2507 - they both did a ton of extra reinforcement learning training related to code. Am I missing something?
- whimsicalism 1y agoI think the primary thing you're missing is that Qwen3-235B-A22B-Instruct-2507 != Qwen3-Coder-480B-A35B-Instruct. And the difference there is that while both do tons of code RL, in one they do not monitor performance on anything else for forgetting/regression and focus fully on code post-training pipelines and it is not meant for other tasks.