4 ms·

LLaMA-13B 4bit needs only 18GB of VRAM for GPT-3 175B level text generation. But your point stands. LLaMA-65B 4bit needs 36 GB of VRAM, but far exceeds GPT-3's

LLaMA-13B 4bit needs only 18GB of VRAM for GPT-3 175B level text generation. But your point stands. LLaMA-65B 4bit needs 36 GB of VRAM, but far exceeds GPT-3's capabilities and even takes on PaLM 540B. See: https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model#4-bit-mode https://github.com/oobabooga/text-generation-webui/wiki/LLaM... for 4bit setup instructions See also: The case for 4-bit precision, which shows effectively no output quality reduction for these 4bit quantization methods (and considerable speedup) https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720

sterlind 4y ago
absolutely amazing. I'm stunned how fast quantization was done. do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations?
- MacsHeadroom 4y ago
  I can't edit my comment now, but it's 30B that needs 18GB of VRAM. LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization. >do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations? Absolutely. The GPTQ paper claims negligible output quality loss with 3-bit quantization. The GPTQ-for-LLaMA repo supports 3-bit quantization and inference. So this extra 25% savings is already possible. As of right now GPTQ-for-LLaMA is using a VRAM hungry attention method. Flash attention will reduce the requirements for 7B to 4GB and possibly fit 30B with a 2048 context window into 16GB, all before stacking 3-bit. Pruning is a possibility but I'm not aware of anyone working on it yet. LoRa has already been implemented. See https://github.com/zphang/minimal-llama#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...
Der_Einzige 4y ago
Effective 4bit quantization seems like evidence of the devil's existence to me. This shouldn't be possible.

sterlind 4y ago

absolutely amazing. I'm stunned how fast quantization was done. do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations?

MacsHeadroom 4y ago
I can't edit my comment now, but it's 30B that needs 18GB of VRAM. LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization. >do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations? Absolutely. The GPTQ paper claims negligible output quality loss with 3-bit quantization. The GPTQ-for-LLaMA repo supports 3-bit quantization and inference. So this extra 25% savings is already possible. As of right now GPTQ-for-LLaMA is using a VRAM hungry attention method. Flash attention will reduce the requirements for 7B to 4GB and possibly fit 30B with a 2048 context window into 16GB, all before stacking 3-bit. Pruning is a possibility but I'm not aware of anyone working on it yet. LoRa has already been implemented. See https://github.com/zphang/minimal-llama#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...

Der_Einzige 4y ago

Effective 4bit quantization seems like evidence of the devil's existence to me. This shouldn't be possible.