MOE models are trained all at once, they're not simply ensembling already trained models. Also data quality and quantity matter considerably, and how OpenAI get

MOE models are trained all at once, they're not simply ensembling already trained models. Also data quality and quantity matter considerably, and how OpenAI gets their data is not public