3 ms·

> Everyone groups all the requests into a batch, and the GPU computes them together. You're only saving on fetching read-only parameters, and not even on that

> Everyone groups all the requests into a batch, and the GPU computes them together. You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV-cache, activations - gets multiplied by your batch size. You scale both compute and memory pressure by largely the same amount. Yes, GPUs are great at hiding memory fetch latency, but that applies also to n=1 inference.

jychang 8mo ago
Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth. I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.
- zozbot234 8mo ago
  > Well, the actual inference providers put each expert on its own single GPU. Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
  - jychang 8mo ago
    Correct, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320 means expert parallelism, each on 320 GPUs.

jychang 8mo ago

Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth. I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.

zozbot234 8mo ago
> Well, the actual inference providers put each expert on its own single GPU. Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
- jychang 8mo ago
  Correct, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320 means expert parallelism, each on 320 GPUs.