3 ms·
> a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. What kind of hardware do you need to run this?
by janice1999 2y ago
> a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.
What kind of hardware do you need to run this?
- boroboro4 2y agoThey discuss it in the paper and recommend 32 GPUs (H800 in their case) for prefill stage and 320 GPUs for decoding. =)
- bavell 2y ago8x H200s recommended: https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3 https://github.com/sgl-project/sglang/tree/main/benchmark/de...