2 ms·
its moe, each expert tower can be branched from some smaller model.
by riku_iki 1y ago
its moe, each expert tower can be branched from some smaller model.
- jychang 1y agoThat's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.