4 ms·
Before each feed forward block, there is a single linear layer outputting 8 numbers that tell you how promising each 'expert' is
by sebzim4500 3y ago
Before each feed forward block, there is a single linear layer outputting 8 numbers that tell you how promising each 'expert' is
- mckirk 3y agoSo basically there's a 'consultant' layer.