3 ms·
Is it possible to run the smallest one on a consumer gpu with 24gb ram ?
by mlboss 4y ago
Is it possible to run the smallest one on a consumer gpu with 24gb ram ?
- Tepix 4y agoRunning it is easy but you'll probably want to finetune it, too
- rihegher 4y agoI would be surprised if you can't. The smallest weight file is 14gb apparently
- eightysixfour 4y agohttps://github.com/facebookresearch/llama/blob/main/FAQ.md#3 https://github.com/facebookresearch/llama/blob/main/FAQ.md#3 Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.
- MacsHeadroom 4y agoIn int8 7B needs only 9GB of VRAM and 13B needs only 20GB on a single GPU. https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1454841458 https://github.com/oobabooga/text-generation-webui/issues/14...
- MacsHeadroom 4y agoYou can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1454841458 https://github.com/oobabooga/text-generation-webui/issues/14...