"Expert" in MoE has no bearing on what you might think of as a human expert. It's not like there is one expert that is proficient at science, and one that is p

"Expert" in MoE has no bearing on what you might think of as a human expert. It's not like there is one expert that is proficient at science, and one that is proficient in history. For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.