Exactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the

Exactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.