3 ms·

Excuse my ignorance, but what is a “dense” model? Why is comparing GPT-OSS 20B against 8B dense models “fair” but not when comparing it against 32B models?

jychang 1y ago
So gpt-oss-20b is a sparse MoE model Which means it has ~3b parameters active per token. Qwen3-32b has 32b params active per token
spwa4 1y ago
An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model. Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B Performance comparison: Qwen-32B => 56 tok/sec, 32 GB of VRAM Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory. So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model. Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you. There's even a name for the practice "ensemble models". MoE only kind-of fits (because you can't easily swap out models) There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...

jychang 1y ago

So gpt-oss-20b is a sparse MoE model Which means it has ~3b parameters active per token. Qwen3-32b has 32b params active per token

spwa4 1y ago

An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model. Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B Performance comparison: Qwen-32B => 56 tok/sec, 32 GB of VRAM Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory. So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model. Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you. There's even a name for the practice "ensemble models". MoE only kind-of fits (because you can't easily swap out models) There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...