Unlike what their names imply, MoE don't have a concept of domain expert at all. It's just a fancy name for sparse evaluation of the total network to save comp

Unlike what their names imply, MoE don't have a concept of domain expert at all. It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.