I can't edit my comment now, but it's 30B that needs 18GB of VRAM. LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization. >do y

I can't edit my comment now, but it's 30B that needs 18GB of VRAM. LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization. >do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations? Absolutely. The GPTQ paper claims negligible output quality loss with 3-bit quantization. The GPTQ-for-LLaMA repo supports 3-bit quantization and inference. So this extra 25% savings is already possible. As of right now GPTQ-for-LLaMA is using a VRAM hungry attention method. Flash attention will reduce the requirements for 7B to 4GB and possibly fit 30B with a 2048 context window into 16GB, all before stacking 3-bit. Pruning is a possibility but I'm not aware of anyone working on it yet. LoRa has already been implemented. See https://github.com/zphang/minimal-llama#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...