To be fair, it's a common misconception because of the name and also how it's traditionally explained. It generally gets lost in translation that MoE trains all

To be fair, it's a common misconception because of the name and also how it's traditionally explained. It generally gets lost in translation that MoE trains all the models at once, from the same dataset. I believe each expert also can't generally be used entirely outside of the context of the mixture (i.e. the operation of the MoE relies on picking the right expert to use for each token, even if num_experts is 1), but I haven't seen a definitive answer on this.