3 ms·
Mixture of experts: Make each model world-class within a single domain. If adding one more common-sense QnA makes the calculus-bot even slightly worse at caculu
by icandoit 3y ago
Mixture of experts: Make each model world-class within a single domain.
If adding one more common-sense QnA makes the calculus-bot even slightly worse at caculus, don't do it.
https://en.wikipedia.org/wiki/Mixture_of_experts https://en.wikipedia.org/wiki/Mixture_of_experts
- rjvs 3y agoThe “mixture of experts” concept in LLMs is a way of training a single model, it’s not based on training many different models (although that was the idea when the term was originally coined).