4 ms·
Sounds like you're describing mixture of experts, the architecture being used in openai's gpt-4 and mistral's mixtral series of models.
by laborcontract 2y ago
Sounds like you're describing mixture of experts, the architecture being used in openai's gpt-4 and mistral's mixtral series of models.
- pants2 2y agoNot really, MoE is trained all at once and the 'experts' don't have pre-defined specializations. They end up being more like "punctuation expert" and "pronoun expert" than "math expert" and "french expert"
- laborcontract 2y agoHaven't tried any yet, but it sounds like parent may be interested in an LLM router. https://github.com/lm-sys/RouteLLM https://github.com/lm-sys/RouteLLM