I think the reason OpenAI and Mistral go for this approach is that they are compute constrained when serving their API from the cloud. My guess is that they hav

I think the reason OpenAI and Mistral go for this approach is that they are compute constrained when serving their API from the cloud. My guess is that they have servers with e.g. one A100 per “expert”, and then they load this up with concurrent independent requests until those A100s are all pretty busy. EDIT: In a cloud environment with independent concurrent requests MoE also reduces VRAM requirements because you don’t need to keep as many activations in memory.