3 ms·
> dynamic router for LLMs This is sort of how Mixture-of-Experts models work, actually.
by dheera 2y ago
> dynamic router for LLMs
This is sort of how Mixture-of-Experts models work, actually.
- danlenton 2y agoYep! Although MoE use several "expert" linear layers within a single network, and generally the "routing" is not based on high-level semantics, but token specialization, as disuccussed by Fuzhao Xue, author of OpenMoE, in one of our reading groups: https://www.youtube.com/watch?v=k3QOpJA0A0Q&t=1547s https://www.youtube.com/watch?v=k3QOpJA0A0Q&t=1547s