3 ms·
What's MoE?
by slekker 1y ago
What's MoE?
- zamalek 1y agoMixture of Experts. Very broadly speaking, there are a bunch of mini networks (experts) which can be independently activated.
- Havoc 1y agoMixture of experts like other guy said - everything gets loaded into mem but not every byte is needed to generate a token (unlike classic LLMs like gemma). So for devices that have lots of mem but weaker processing power it can get you similar output quality but faster. So tends to do better on CPU and APU like setups
- trebligdivad 1y agoI'm not even sure they're loading everything into memory for MoE; maybe they can get away with only the relevant experts being paged in.