4 ms·
How does MoE choose an expert? From the outside and if we squint a bit; this looks a lot like an inverted attention mechanism where the token attends to the ex
by PartiallyTyped 2y ago
How does MoE choose an expert?
From the outside and if we squint a bit; this looks a lot like an inverted attention mechanism where the token attends to the experts.
- telotortium 2y agoUsually there’s a small neural network that makes the choice for each token in an LLM.
- HarHarVeryFunny 2y agoI don't know the details, but there are a variety of routing mechanisms that have been tried. One goal is to load balance tokens among the experts so that each expert's parameters are equally utilized, which it seems must sometimes conflict with wanting to route to an expert based on the token itself.
- magicalhippo 2y agoFrom what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware. [1]: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538 [2]: https://arxiv.org/abs/2208.02813 https://arxiv.org/abs/2208.02813 [3]: https://news.ycombinator.com/item?id=40675577 https://news.ycombinator.com/item?id=40675577