3 ms·
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
by why_only_15 3y ago
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
- ShamelessC 3y agoOh I’m happy to admit if I’m wrong in the details. My bad. So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?