4 ms·
In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comp
by rishabhjain1198 3y ago
In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comparable 14B model.
- nullc 3y agoOnly assuming it's able to hide the faster one in free parallelism.
- moffkalast 3y agoMy CPU trying its best to run inference: parallelwhat?