3 ms·
It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token)
by qeternity 2y ago
It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token).
- regularfry 2y agoWeirdly it does have to do with words, but not intentionally. Mechanically the routing is per-token, but the routing is frequently stable across a word as an emergent property. At least, that's how I read the mixtral paper.
- ljlolel 2y agoYep. Also note that I’m ELI5 so saying word is fine.