3 ms·
3T is impressive, but parameter count seems to be less important than I thought. GLM is half the size of DeepSeek but costs four times as much, and beats it on
by andai 4mo ago
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
- pietz 4mo agoIt's almost like they priced models based on their performance or something...
- jnwatson 4mo agoAfter MoE entered the mix, raw parameter count is less useful a measure.
- wolttam 4mo agoOr, GLM 5.2 simply had more time in the RL oven. Deepseek V4 Flash, the 284B model, is roughly equivalent to launch GLM 5, the 744B [sic] model.
- esafak 4mo agoYou have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
- Creamsicle47 4mo ago> You have to look at the size of each expert Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets. > The number of the experts tells you about the diversity of its skills. Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
- dannyw 4mo agoYeah, "experts" is a ML/research word for this (MoE was first published in 1991; and has been around for a long time, it even predates deep learning). it's not the everyday/colloquial meaning of 'expert'.