3 ms·
Weirdly it does have to do with words, but not intentionally. Mechanically the routing is per-token, but the routing is frequently stable across a word as an e
by regularfry 2y ago
Weirdly it does have to do with words, but not intentionally. Mechanically the routing is per-token, but the routing is frequently stable across a word as an emergent property. At least, that's how I read the mixtral paper.
- ljlolel 2y agoYep. Also note that I’m ELI5 so saying word is fine.