3 ms·
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used?
by robrenaud 24d ago
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
- gsora 24d agoIsn't that the definition of an MoE model?
- jmolinski 24d agoNo, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
- greazy 24d agoWhat is "sticky" in this context?
- robrenaud 24d agoExperts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth. If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.