4 ms·
> MoE tends to change expert every other word Any citation on this one?
by htrp 2y ago
> MoE tends to change expert every other word
Any citation on this one?
- crystal_revenge 2y agoIt's covered in the original Mistral "Mixtral of Experts" paper [0]. 0. https://arxiv.org/abs/2401.04088 https://arxiv.org/abs/2401.04088
- Ey7NFZ3P0nzAe 2y agoI believe it's actually a per token routing, not a "every few words"
- regularfry 2y agoIt's mechanically capable of per-token routing, but the routing tends to be stable across more than one token. It's weird.