5 ms·

The terminal bench scores look weak but nice otherwise. I hope once the benchmarks are saturated, companies can focus on shrinking the models. Until then, let t

by esafak 11mo ago

The terminal bench scores look weak but nice otherwise. I hope once the benchmarks are saturated, companies can focus on shrinking the models. Until then, let the games continue.

CuriouslyC 11mo ago
We're not gonna see significant model shrinkage until the money tap dries up. Between now and then, we'll see new benchmarks/evals that push the holes in model capabilities in cycles as they saturate each new round.
- lanthissa 11mo ago
  isn't gemini 3 flash already model shrinkage that does well in coding?
  - hedgehog 11mo ago
    Smaller open-weights models are also improving noticeably (like Qwen3 Coder 30B), the improvements are happening at all sizes.
    - cmrdporcupine 11mo ago
      Devstral Small 24b looks promising as something I want to try fine tuning on DSLs, etc. and then embedding in tooling.
      - hedgehog 11mo ago
        I haven't tried it yet, but yes. Qwen3 Next 80B works decently in my testing, and fast. I had mixed results with the new Nemotron, but it and the new Qwen models are both very fast to run.
        mark_l_watson 11mo ago
        Same experience: on my old M2 Mac with just 32B of memory both Qwen 3 30B and the new Nemotron models are very useful for coding if I prepare a one-shot prompt with directions and relevant code. I don’t like them for agentic coding tools. I have mentioned this elsewhere: it is deeply satisfying to mix local model use with commercial APIs and services.
  - Imustaskforhelp 11mo ago
    How much billion parameter model is gemini 3 flash, I can't seem to find info about it online.
  - skippyboxedhero 11mo ago
    Xiaomi, Nvidia Nemotron, Minimax, lots of other smaller ones too. There are massive economic incentives to shrink models because they can be provided faster and at lower cost. I think even with the money going in, there has to be some revenue supporting that development somewhere. And users are now looking at the cost. I have been using Anthropic Max for most of this year after checking out some of these other models, it is clearly overpriced (I would also say their moat of Claude Code has been breached). And Anthropic's API pricing is completely crazy when you use some of the paradigms that they suggest (agents/commands/etc) i.e. token usage is going up so efficient models are driving growth.
- naasking 11mo ago
  > We're not gonna see significant model shrinkage until the money tap dries up. I'm not sure about that. Microsoft has been doing great work on "1-bit" LLMs, and dropping the memory requirements would significantly cut down on operating costs for the frontier players.
bigyabai 11mo ago
It's a good model, for what it is. Z.ai's big business prop is that you can get Claude Code with their GLM models at much lower prices than what Anthropic charges. This model is going to be great for that agentic coding application.
- maxdo 11mo ago
  … and wake up every night because you saved a few dollars , there are bugs and they are due to this decision?
  - Imustaskforhelp 11mo ago
    well I feel like all models are converging and maybe claude is good but only time will tell as gemini flash and GLM put pressure on claude/anthropic models People (here) are definitely comparing it to sonnet so if you take this stance of saving a few dollars, I am sure that you must be having the same opinion of using opus model and nobody should use sonnet too Personally I am interested in open source models because they would be something which would have genuine value and competition after the bubble bursts
  - bigyabai 11mo ago
    I pay for both Claude and Z.ai right now, and GLM-4.7 is more than capable for what I need. Opus 4.5 is nice but not worth the quota cost for most tasks.
  - csomar 11mo ago
    Yeah because Claude never makes bugs?
theshrike79 11mo ago
z.ai models are crazy cheap. The one year lite plan is like 30€ (on sale though). Complete no-brainer to get it as a backup with Crush. I've been using it for read-only analysis and implementing already planned tasks with pretty good results. It has a slight habit of expanding scope without being asked. Sometimes it's a good thing, sometimes it does useless work or messes things up a bit.
- maxdo 11mo ago
  I tried several times . It is no match in my personal experience with Claude models . There’s almost no place for second spot from my point of view . You are doing things for work each bug is hours of work, potentially lost customer etc . Why would you trust your money … just to back up ?
  - theshrike79 11mo ago
    I'm using it for my own stuff and I'm definitely not dropping however much it costs for the Claude Max plans. That's why I usually use Claude for planning, feed the issues to beads or a markdown file and then have Codex or Crush+GLM implement them. For exploratory stuff I'm "pair-programming" with Claude. At work we have all the toys, but I'm not putting my own code through them =)
    - maxdo 11mo ago
      it's beyond me, why do you need Max plans? I use Opus/Sonnet/Gemini,GPT 5.2 every day in cursor and I'm not paying Claude Max.
      - theshrike79 11mo ago
        I'm mostly just coding at night after the family goes to bed and even I can hit Claude Pro limits - and I started AI assisted programming when we didn't have monthly plans and I had to pay every token out of my own pocket. I learned to be pretty efficient with token use after the first bill dropped :D
  - 11mo ago
anonzzzies 11mo ago
Shrinking and speed; speed is a major thing. Claude Code is just too slow, very good but it has no reasonable way to handle simple requests because of the overhead, so then everything should just be faster. If I were Anthropic, I would've bought Groq or Cerebras by now. Not sure if they (or the other big ones) are working on similar inference hardware to provide 2000tok/s or more.
- pqtyw 11mo ago
  Z.ai (at least mid/top end subscription not sure about the API) is pretty slow too especially during some periods. Cerebras of course is probably a different story (if its not quantitized)

by esafak 11mo ago

The terminal bench scores look weak but nice otherwise. I hope once the benchmarks are saturated, companies can focus on shrinking the models. Until then, let the games continue.

CuriouslyC 11mo ago

We're not gonna see significant model shrinkage until the money tap dries up. Between now and then, we'll see new benchmarks/evals that push the holes in model capabilities in cycles as they saturate each new round.

lanthissa 11mo ago
isn't gemini 3 flash already model shrinkage that does well in coding?
- hedgehog 11mo ago
  Smaller open-weights models are also improving noticeably (like Qwen3 Coder 30B), the improvements are happening at all sizes.
  - cmrdporcupine 11mo ago
    Devstral Small 24b looks promising as something I want to try fine tuning on DSLs, etc. and then embedding in tooling.
    - hedgehog 11mo ago
      I haven't tried it yet, but yes. Qwen3 Next 80B works decently in my testing, and fast. I had mixed results with the new Nemotron, but it and the new Qwen models are both very fast to run.
      - mark_l_watson 11mo ago
        Same experience: on my old M2 Mac with just 32B of memory both Qwen 3 30B and the new Nemotron models are very useful for coding if I prepare a one-shot prompt with directions and relevant code. I don’t like them for agentic coding tools. I have mentioned this elsewhere: it is deeply satisfying to mix local model use with commercial APIs and services.
- Imustaskforhelp 11mo ago
  How much billion parameter model is gemini 3 flash, I can't seem to find info about it online.
- skippyboxedhero 11mo ago
  Xiaomi, Nvidia Nemotron, Minimax, lots of other smaller ones too. There are massive economic incentives to shrink models because they can be provided faster and at lower cost. I think even with the money going in, there has to be some revenue supporting that development somewhere. And users are now looking at the cost. I have been using Anthropic Max for most of this year after checking out some of these other models, it is clearly overpriced (I would also say their moat of Claude Code has been breached). And Anthropic's API pricing is completely crazy when you use some of the paradigms that they suggest (agents/commands/etc) i.e. token usage is going up so efficient models are driving growth.
naasking 11mo ago
> We're not gonna see significant model shrinkage until the money tap dries up. I'm not sure about that. Microsoft has been doing great work on "1-bit" LLMs, and dropping the memory requirements would significantly cut down on operating costs for the frontier players.

bigyabai 11mo ago

It's a good model, for what it is. Z.ai's big business prop is that you can get Claude Code with their GLM models at much lower prices than what Anthropic charges. This model is going to be great for that agentic coding application.

maxdo 11mo ago
… and wake up every night because you saved a few dollars , there are bugs and they are due to this decision?
- Imustaskforhelp 11mo ago
  well I feel like all models are converging and maybe claude is good but only time will tell as gemini flash and GLM put pressure on claude/anthropic models People (here) are definitely comparing it to sonnet so if you take this stance of saving a few dollars, I am sure that you must be having the same opinion of using opus model and nobody should use sonnet too Personally I am interested in open source models because they would be something which would have genuine value and competition after the bubble bursts
- bigyabai 11mo ago
  I pay for both Claude and Z.ai right now, and GLM-4.7 is more than capable for what I need. Opus 4.5 is nice but not worth the quota cost for most tasks.
- csomar 11mo ago
  Yeah because Claude never makes bugs?

theshrike79 11mo ago

z.ai models are crazy cheap. The one year lite plan is like 30€ (on sale though). Complete no-brainer to get it as a backup with Crush. I've been using it for read-only analysis and implementing already planned tasks with pretty good results. It has a slight habit of expanding scope without being asked. Sometimes it's a good thing, sometimes it does useless work or messes things up a bit.

maxdo 11mo ago
I tried several times . It is no match in my personal experience with Claude models . There’s almost no place for second spot from my point of view . You are doing things for work each bug is hours of work, potentially lost customer etc . Why would you trust your money … just to back up ?
- theshrike79 11mo ago
  I'm using it for my own stuff and I'm definitely not dropping however much it costs for the Claude Max plans. That's why I usually use Claude for planning, feed the issues to beads or a markdown file and then have Codex or Crush+GLM implement them. For exploratory stuff I'm "pair-programming" with Claude. At work we have all the toys, but I'm not putting my own code through them =)
  - maxdo 11mo ago
    it's beyond me, why do you need Max plans? I use Opus/Sonnet/Gemini,GPT 5.2 every day in cursor and I'm not paying Claude Max.
    - theshrike79 11mo ago
      I'm mostly just coding at night after the family goes to bed and even I can hit Claude Pro limits - and I started AI assisted programming when we didn't have monthly plans and I had to pay every token out of my own pocket. I learned to be pretty efficient with token use after the first bill dropped :D
- 11mo ago

anonzzzies 11mo ago

Shrinking and speed; speed is a major thing. Claude Code is just too slow, very good but it has no reasonable way to handle simple requests because of the overhead, so then everything should just be faster. If I were Anthropic, I would've bought Groq or Cerebras by now. Not sure if they (or the other big ones) are working on similar inference hardware to provide 2000tok/s or more.

pqtyw 11mo ago
Z.ai (at least mid/top end subscription not sure about the API) is pretty slow too especially during some periods. Cerebras of course is probably a different story (if its not quantitized)