3 ms·What 1T parameter base model have you seen from any of those labs?by zackangelo 11mo agoWhat 1T parameter base model have you seen from any of those labs?riku_iki 11mo agoits moe, each expert tower can be branched from some smaller model.jychang 11mo agoThat's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.