3 ms·

The new Meta 30B models seems A LOT better: https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvidia-nemotron-3-5-lightning-high/ https://aibenchy.com/c

by XCSme 2mo ago

The new Meta 30B models seems A LOT better: https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvidia-nemotron-3-5-lightning-high/ https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...

Tactical45 2mo ago
At what cost difference?
- XCSme 2mo ago
  I don't think it matters, if it's for local/on-device usage. The cost is similar vram footprint I guess (?)
  - sleepyeldrazi 2mo ago
    loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test). As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
    - XCSme 2mo ago
      I am asking mostly for running on a 3090. I think the tps difference between them (both fitting in vram) won't be more than 2x in practice. I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
      - sleepyeldrazi 2mo ago
        Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params). You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
        XCSme 2mo ago
        I remember running both qwen 30b-a3b and 27b on my 3090, and on the initial test, the 27b was only like 2x slower.
        sleepyeldrazi 2mo ago
        Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference. For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
        XCSme 2mo ago
        Thanks for sharing. Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention). What was the result with MTP? Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
eli 2mo ago
The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.
- XCSme 2mo ago
  I should add a F.a.q. for this question. The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes. Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice. But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
- markasoftware 2mo ago
  yep, the person you're responding to created the benchmark and is using HN comments as advertisement.
  - XCSme 2mo ago
    I did, avertisment is a big word, as I gain nothing from the traffic, I run the website for myself, and some other people find it useful too. Happy to hear what would make the website more useful.
    - gaflo 2mo ago
      You are displaying ads on your website, there's a clear financial incentive.
      - XCSme 2mo ago
        True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it. Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful. I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
thehamkercat 2mo ago
Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old) but Qwen 3.8 27B is dropping this week...
- XCSme 2mo ago
  Yes, I was surprised to see doing it as well as Qwen 3.7 27b. Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model. Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
khimaros 2mo ago
lightning is sparse, glimmer is dense
- XCSme 2mo ago
  Oh, good to know, I just quickly tested and published the results. I will add model sizes (total/active params) for each model, good point.
rllearneratwork 2mo ago
and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW
- XCSme 2mo ago
  Is that the case? If the entire model fits in vram, won't the tps be comparable?
  - option 2mo ago
    autoregressive generation is memory bandwidth bounded
    - XCSme 2mo ago
      So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps? In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.

by XCSme 2mo ago

Tactical45 2mo ago

At what cost difference?

XCSme 2mo ago
I don't think it matters, if it's for local/on-device usage. The cost is similar vram footprint I guess (?)
- sleepyeldrazi 2mo ago
  loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test). As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
  - XCSme 2mo ago
    I am asking mostly for running on a 3090. I think the tps difference between them (both fitting in vram) won't be more than 2x in practice. I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
    - sleepyeldrazi 2mo ago
      Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params). You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
      - XCSme 2mo ago
        I remember running both qwen 30b-a3b and 27b on my 3090, and on the initial test, the 27b was only like 2x slower.
        sleepyeldrazi 2mo ago
        Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference. For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
        XCSme 2mo ago
        Thanks for sharing. Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention). What was the result with MTP? Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.

eli 2mo ago

The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.

XCSme 2mo ago
I should add a F.a.q. for this question. The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes. Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice. But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
markasoftware 2mo ago
yep, the person you're responding to created the benchmark and is using HN comments as advertisement.
- XCSme 2mo ago
  I did, avertisment is a big word, as I gain nothing from the traffic, I run the website for myself, and some other people find it useful too. Happy to hear what would make the website more useful.
  - gaflo 2mo ago
    You are displaying ads on your website, there's a clear financial incentive.
    - XCSme 2mo ago
      True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it. Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful. I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?

thehamkercat 2mo ago

Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old) but Qwen 3.8 27B is dropping this week...

XCSme 2mo ago
Yes, I was surprised to see doing it as well as Qwen 3.7 27b. Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model. Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.

khimaros 2mo ago

lightning is sparse, glimmer is dense

XCSme 2mo ago
Oh, good to know, I just quickly tested and published the results. I will add model sizes (total/active params) for each model, good point.

rllearneratwork 2mo ago

and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW

XCSme 2mo ago
Is that the case? If the entire model fits in vram, won't the tps be comparable?
- option 2mo ago
  autoregressive generation is memory bandwidth bounded
  - XCSme 2mo ago
    So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps? In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.