3 ms·
This is the first thought came to my mind too. Given its sparse, Will this be just replacement for MoE.
by gautam5669 2y ago
This is the first thought came to my mind too.
Given its sparse, Will this be just replacement for MoE.
- samus 2y agoMoE is mostly used to enable load balancing since it makes it possible to put experts on different GPUs. This isn't so easy to do with a monolithic, but sparse layer.