3 ms·
But the RAM+VRAM can never be less than the size of the total (not active) model, right?
by theanonymousone 1y ago
But the RAM+VRAM can never be less than the size of the total (not active) model, right?
- NitpickLawyer 1y agoCorrect. You want everything loaded, but for each forward pass just some experts get activated so the computation is less than in a dense model. That being said, there are libraries that can load a model layer by layer (say from an ssd) and technically perform inference with ~8gb of RAM, but it'd be really really slow.
- theanonymousone 1y agoCan you give me a name please? Is that distributed llama or something else?
- skirmish 1y agoI have not used it but this is probably it: https://github.com/lyogavin/airllm https://github.com/lyogavin/airllm