3 ms·
I think that’s only true for MoE models. A dense model like 3.6 27b will require more (plus a KV store).
by jnovek 4mo ago
I think that’s only true for MoE models. A dense model like 3.6 27b will require more (plus a KV store).
- bityard 4mo agoNo, even MoE models need to fit into (V)RAM. MoE has faster inference because only a subset of layers are used to predict the next token, but the set of layers used changes with every token.