3 ms·
"Mixture of Expert (MoE) models rely on multiple optimized smaller models, each with a more refined and specified dataset" This is not how MoEs work...
by qeternity 2y ago
"Mixture of Expert (MoE) models rely on multiple optimized smaller models, each with a more refined and specified dataset"
This is not how MoEs work...
- LoganDark 2y agoTo be fair, it's a common misconception because of the name and also how it's traditionally explained. It generally gets lost in translation that MoE trains all the models at once, from the same dataset. I believe each expert also can't generally be used entirely outside of the context of the mixture (i.e. the operation of the MoE relies on picking the right expert to use for each token, even if num_experts is 1), but I haven't seen a definitive answer on this.
- qeternity 2y agoI don't know who is traditionally explaining MoEs like this but they shouldn't be. And especially for someone writing an article on technical LLM operation...the author just clearly lacks the domain expertise that I would expect.