3 ms·
Can't visualize the graphs well from mobile but the many layers are there because MoE models select a different expert at each layer, not just at each token.
by antirez 3y ago
Can't visualize the graphs well from mobile but the many layers are there because MoE models select a different expert at each layer, not just at each token.