3 ms·
Quick example, Kimi K2 is a recent large mixture of experts model. Each “expert” is really just a path within it. At each token, 32B out of 1T are active. This
by janalsncm 1y ago
Quick example, Kimi K2 is a recent large mixture of experts model. Each “expert” is really just a path within it. At each token, 32B out of 1T are active. This means only 3.2% are active for any one token.
- Sophira 1y agoThat sounds surprisingly like "Humans only use 10% of their brain at any given time."