4 ms·
Anyone ever look at doing a MoE like composition with RWKV and a transformer?
by nullc 2y ago
Anyone ever look at doing a MoE like composition with RWKV and a transformer?
- pico_creator 2y agoNot an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget) https://arxiv.org/abs/2407.12077 https://arxiv.org/abs/2407.12077