3 ms·
I you look at https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token
by kristianp 3mo ago
I you look at https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
- xyzsparetimexyz 3mo agoFor purely coding tasks, is every single expert required? 50%? 25%?
- FeepingCreature 3mo agoYes, every expert. It's not like different personas, more like very coarse dropout training.
- xyzsparetimexyz 3mo agoRight. So it's not like human experts or even different brain regions