Isn't MoE with switch transformers massively inefficiemt compared to being able to customize which LLMs you are using? I've seen a lot of agent swarm concepts

Isn't MoE with switch transformers massively inefficiemt compared to being able to customize which LLMs you are using? I've seen a lot of agent swarm concepts in the smaller llm space that seem to provide some feedback that this is a viable avenue of research.