3 ms·

Why is it so small, but expensive? Open Router Input /M $0.45 Output /M $3.20 Cache read /M $0.05 Throughput 27 tps It would be a very nice model at 200-3

by f311a 2mo ago

Why is it so small, but expensive? Open Router Input /M $0.45 Output /M $3.20 Cache read /M $0.05 Throughput 27 tps It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?

theanonymousone 2mo ago
That's my questions as well. DeepSeek v4 0731 is served dirt cheap and it needs 10 times more RAM.
- kmike84 2mo ago
  DeepSeek needs more RAM for weights, Qwen requires more compute. Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.
- petu 2mo ago
  > and it needs 10 times more RAM. More like 3-6. Qwen 27B full quality is FP16. So 54GB. In practice most inference providers would serve FP8, so 27GB. DeepSeek V4 Flash in full quality is mostly FP4. ~167GB official release. So Deepseek has 140GB model size overhead... which is shared between 100s of users single inference node serves, so not even a gigabyte of VRAM per user. Memory required for 200K of context per user: V4 Flash: 1GB. Qwen 27B: 13GB.
freakynit 2mo ago
I read it somewhere recently that it's architecture does not allow serving as many concurrent requests as the deepseek models allow. Maybe that's why.
- FuckButtons 2mo ago
  From first principles, 27b dense vs 13b moe, means you spend ~2x more memory bandwidth per request amortized over the whole server (ie, you assume all experts are being concurrently used by some user during the forward pass, then on average the bandwidth required for one forward pass for any individual request is just the size of one expert). deepseek also have some innovations around kv cache and compressed attention which allow for further reductions in memory bandwidth which means that the thing that’s actually bottlenecking inference, (memory bandwidth) is significantly lower than for qwen 3.8, which has been optimized for running 1 instance ~= 1 user.
  - meatmanek 2mo ago
    When processing multiple users in parallel, don't you end up having to load in multiple experts? Not every session is going to use each expert at exactly the same time.
- Grimblewald 2mo ago
  where'd you read that? sounds like total bs but i could be wrong and would like to learn more.
  - freakynit 2mo ago
    Read this on Reddit.. maybe it's total bs.. maybe not... i dont know.. mentioned in case someone wants to dig deeper into this.
thenews 2mo ago
more profits i guess, https://news.ycombinator.com/item?id=49323381 https://news.ycombinator.com/item?id=49323381
AgentLemon 2mo ago
It's a dense model, so 27B active parameters to compute. Compare that to DeepSeek V4 Flash, which has only 13B active parameters (MoE).
anotherCodder 2mo ago
[dead]
tipsytoad 2mo ago
It’s 27b active params vs 13b active params, so you’d expect it to be 2x more expensive when serving multiple users

by f311a 2mo ago

theanonymousone 2mo ago

That's my questions as well. DeepSeek v4 0731 is served dirt cheap and it needs 10 times more RAM.

kmike84 2mo ago
DeepSeek needs more RAM for weights, Qwen requires more compute. Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.
petu 2mo ago
> and it needs 10 times more RAM. More like 3-6. Qwen 27B full quality is FP16. So 54GB. In practice most inference providers would serve FP8, so 27GB. DeepSeek V4 Flash in full quality is mostly FP4. ~167GB official release. So Deepseek has 140GB model size overhead... which is shared between 100s of users single inference node serves, so not even a gigabyte of VRAM per user. Memory required for 200K of context per user: V4 Flash: 1GB. Qwen 27B: 13GB.

freakynit 2mo ago

I read it somewhere recently that it's architecture does not allow serving as many concurrent requests as the deepseek models allow. Maybe that's why.

FuckButtons 2mo ago
From first principles, 27b dense vs 13b moe, means you spend ~2x more memory bandwidth per request amortized over the whole server (ie, you assume all experts are being concurrently used by some user during the forward pass, then on average the bandwidth required for one forward pass for any individual request is just the size of one expert). deepseek also have some innovations around kv cache and compressed attention which allow for further reductions in memory bandwidth which means that the thing that’s actually bottlenecking inference, (memory bandwidth) is significantly lower than for qwen 3.8, which has been optimized for running 1 instance ~= 1 user.
- meatmanek 2mo ago
  When processing multiple users in parallel, don't you end up having to load in multiple experts? Not every session is going to use each expert at exactly the same time.
Grimblewald 2mo ago
where'd you read that? sounds like total bs but i could be wrong and would like to learn more.
- freakynit 2mo ago
  Read this on Reddit.. maybe it's total bs.. maybe not... i dont know.. mentioned in case someone wants to dig deeper into this.

thenews 2mo ago

more profits i guess, https://news.ycombinator.com/item?id=49323381 https://news.ycombinator.com/item?id=49323381

AgentLemon 2mo ago

It's a dense model, so 27B active parameters to compute. Compare that to DeepSeek V4 Flash, which has only 13B active parameters (MoE).

anotherCodder 2mo ago

[dead]

tipsytoad 2mo ago

It’s 27b active params vs 13b active params, so you’d expect it to be 2x more expensive when serving multiple users