3 ms·
It's not clear from the article whether it's a dense model or MoE. This matters when it comes to comparing with GPT-4 - in terms of # params - which is reported
by py4 3y ago
It's not clear from the article whether it's a dense model or MoE. This matters when it comes to comparing with GPT-4 - in terms of # params - which is reported to be MoE
- jonnycomputer 3y ago[flagged]
- freeqaz 3y agoWhat is MoE? Edit: Ah, Mixture of Experts. I hadn't heard this one yet. Thanks!
- COAGULOPATH 3y agoAs far as I know, EVERY 1t+ LLM is a MoE. Switch-c-2048, Wi Dao 2.0, GLAM, Pangu-Σ, presumably GPT4. Am I missing any?