3 ms·
I read it somewhere recently that it's architecture does not allow serving as many concurrent requests as the deepseek models allow. Maybe that's why.
by freakynit 2mo ago
I read it somewhere recently that it's architecture does not allow serving as many concurrent requests as the deepseek models allow. Maybe that's why.
- FuckButtons 2mo agoFrom first principles, 27b dense vs 13b moe, means you spend ~2x more memory bandwidth per request amortized over the whole server (ie, you assume all experts are being concurrently used by some user during the forward pass, then on average the bandwidth required for one forward pass for any individual request is just the size of one expert). deepseek also have some innovations around kv cache and compressed attention which allow for further reductions in memory bandwidth which means that the thing that’s actually bottlenecking inference, (memory bandwidth) is significantly lower than for qwen 3.8, which has been optimized for running 1 instance ~= 1 user.
- meatmanek 2mo agoWhen processing multiple users in parallel, don't you end up having to load in multiple experts? Not every session is going to use each expert at exactly the same time.
- Grimblewald 2mo agowhere'd you read that? sounds like total bs but i could be wrong and would like to learn more.
- freakynit 2mo agoRead this on Reddit.. maybe it's total bs.. maybe not... i dont know.. mentioned in case someone wants to dig deeper into this.