2 ms·What 1T parameter base model have you seen from any of those labs?by zackangelo 1y agoWhat 1T parameter base model have you seen from any of those labs?riku_iki 1y agoits moe, each expert tower can be branched from some smaller model.jychang 1y agoThat's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.