Exactly, same thought. Actually I would expect, that they trained each expert separately and later together, since you need to train the router network as well.

Exactly, same thought. Actually I would expect, that they trained each expert separately and later together, since you need to train the router network as well. I'm far from an expert in LLMs. But this would be interesting to know, especially how different training setups influence the performance.