2 ms·
How does Mixtral decide which Expert is involved how much/when?
by tosh 3y ago
How does Mixtral decide which Expert is involved how much/when?
- sebzim4500 3y agoBefore each feed forward block, there is a single linear layer outputting 8 numbers that tell you how promising each 'expert' is
- mckirk 3y agoSo basically there's a 'consultant' layer.