3 ms·
Its an MoE model so only 37B parameters are activated per token.
by enragedcacti 2y ago
Its an MoE model so only 37B parameters are activated per token.
- aurareturn 2y agoMoE models aren’t new. Rumors are that GPT4 was one.
- swyx 2y agoyea specifically 8x220B https://www.latent.space/p/geohot https://www.latent.space/p/geohot
- enragedcacti 2y agoYes, with substantially more active params for SOTA.