From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they use

From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware. [1]: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538 [2]: https://arxiv.org/abs/2208.02813 https://arxiv.org/abs/2208.02813 [3]: https://news.ycombinator.com/item?id=40675577 https://news.ycombinator.com/item?id=40675577