2 ms·

It’s very similar to Mixture of Experts. But instead of routing tokens to multiple experts, you "deploy to a single expert which can be dynamically skipped"

• erikaww 2y ago
  Mixing these would be pretty cool. Further reduced compute for MoE while keeping the performance.
  • GaggiX 2y ago
    In the paper they already show a mixing of these two with Mixture-of-Depths-and-Experts (MoDE).