3 ms·
> There is not way it's sending experts to the GPU per token. Right, it seems like either experts are stable across sequential tokens fairly often, or there's
by furyofantares 1y ago
> There is not way it's sending experts to the GPU per token.
Right, it seems like either experts are stable across sequential tokens fairly often, or there's more than 4 experts in memory and it's stable within the in-memory experts for sequential tokens fairly often, like the poster said.