4 ms·
That's not entirely correct. Most of moe right now are fully balanced, but there is an idea of a domain expert moe where the training benefits fewer switches. h
by viraptor 1y ago
That's not entirely correct. Most of moe right now are fully balanced, but there is an idea of a domain expert moe where the training benefits fewer switches. https://arxiv.org/abs/2410.07490 https://arxiv.org/abs/2410.07490
- idiotsecant 1y agoYes, explicitly trained experts were a thing for a little while, but not anymore. Yet another application of the Hard Lesson.