3 ms·
That's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.
by jychang 11mo ago
That's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.