3 ms·
sorry what does it mean for only 2b to be active?
by apwell23 1y ago
sorry what does it mean for only 2b to be active?
- simonw 1y agoMy understanding is that, while all 8B are loaded into memory, for each token inference step only 2B are selected and used - so tokens are produced faster because there is less computation needed. Hoping someone will correct me if that's not the right mental model!