6 ms·
For sparse MoE models, the single expert layers that the inference gets sampled from are actually quite small - single-digit megabytes or so.
by zozbot234 4mo ago
For sparse MoE models, the single expert layers that the inference gets sampled from are actually quite small - single-digit megabytes or so.