3 ms·
The upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU
by Havoc 2y ago
The upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU
- slekker 2y agoWhat's MoE?
- zamalek 2y agoMixture of Experts. Very broadly speaking, there are a bunch of mini networks (experts) which can be independently activated.
- Havoc 2y agoMixture of experts like other guy said - everything gets loaded into mem but not every byte is needed to generate a token (unlike classic LLMs like gemma). So for devices that have lots of mem but weaker processing power it can get you similar output quality but faster. So tends to do better on CPU and APU like setups
- trebligdivad 2y agoI'm not even sure they're loading everything into memory for MoE; maybe they can get away with only the relevant experts being paged in.