3 ms·
> Well, the actual inference providers put each expert on its own single GPU. Experts are usually chosen on a per-layer basis, not just by token, so I'd think
by zozbot234 8mo ago
> Well, the actual inference providers put each expert on its own single GPU.
Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.
- jychang 8mo agoCorrect, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320 means expert parallelism, each on 320 GPUs.