3 ms·
Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.
by erikaww 3y ago
Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.
- GaggiX 3y agoIn the paper they already show a mixing of these two with Mixture-of-Depths-and-Experts (MoDE).