3 ms·
One key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some
by oezi 2mo ago
One key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some other expert every token? Wouldn't it be more memory efficient to generate at least 2,3,4,8,12 or 32 tokens and then swap the experts?
- zozbot234 2mo agoExpert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly just a way to enforce some kind of model sparsity and save on compute. The new version of Apple Foundation Model (AFM) Core Advanced is an exception, it actually routes experts per prompt (with sporadic rerouting perhaps?) which is more in line with what you're proposing. But this will theoretically make the model less smart than a similar one where experts are picked per layer.