47 ms·
Then what's the advantage of this technique compared to running a +50B model in the first place?
by behnamoh 3y ago
Then what's the advantage of this technique compared to running a +50B model in the first place?
- nulld3v 3y agoThe model is quicker to evaluate. So quicker responses and more throughput.
- hnuser123456 3y agoSpeed, assuming you have the RAM to have it all loaded, it's faster than a fully connected network of the same size by 4x
- rileyphone 3y agoIt's better if you're hosting inference, worse if you are using it for a dedicated purpose. Presumably in the future it might make sense to share one local MoE among the different programs that use it, especially for a demand-heavy application like programming.
- andersa 3y agoIt's faster to run inference and training. Less memory bandwidth needed.