13 ms·
> maybe even learned prefetching for what the next experts will be Experts are predicted by layer and the individual layer reads are quite small, so this is no
by zozbot234 8mo ago
> maybe even learned prefetching for what the next experts will be
Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.
- snovv_crash 8mo agoManually no. It would have to be learned, and making the expert selection predictable would need to be a training metric to minimize.
- zozbot234 8mo agoMaking the expert selection more predictable also means making it less effective. There's no real free lunch.
- yorwba 8mo agoIt's feasible to put the expert routing logic in a previous layer. People have done it: https://arxiv.org/abs/2507.20984 https://arxiv.org/abs/2507.20984