3 ms·
LLaMA Text-Generation-Webui
- MacsHeadroom 4y agotext-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1] LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization. LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B. Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090. *Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention). [0] https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa [1] https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 [2] https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1464346699 https://github.com/oobabooga/text-generation-webui/issues/14... For LLaMA set-up instructions, including GPTQ 4bit, refer to this wiki article: https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model https://github.com/oobabooga/text-generation-webui/wiki/LLaM...
- Tepix 4y agoSo, is anyone doing open source work on finetuning this model? Can finetuning be done with the 4bit weights?
- MacsHeadroom 4y agoClose, fine tuning LLaMA in 8bit has been done. It requires very little VRAM compared to typical fine tuning. https://github.com/zphang/minimal-llama#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...
- KaoruAoiShiho 4y agoIs 2x4090 possible or just 3090?
- MacsHeadroom 4y agoYes, text-generation-webui has multiGPU support regardless of your GPU(s). They do not even need to be the same GPU. You can have an AMD 6900xt combined with an RTX 4090, combined with an RTX 3050, if you want. It makes no difference.
- webmaven 4y agoNote that the notebook supports GPT-J 6B, OPT, GALACTICA, and Pygmalion, not just LLaMA.