3 ms·
It is. I have 48GB of VRAM. But exl2 is more efficient, and can be quantized to partial bits. So you can run things like 4.75bpw, etc. I can run 120b models at
by idonotknowwhy 3y ago
It is. I have 48GB of VRAM. But exl2 is more efficient, and can be quantized to partial bits. So you can run things like 4.75bpw, etc.
I can run 120b models at 3bpw.
The larger models like this are less affected (increased perplexity) by the quantization.
- ComputerGuru 3y agoDid you have to quantize it yourself to 4.75bpw and 3bpw or are they readily available for download?
- idonotknowwhy 3y agoMost of the time it's readily available eg: Panchovix/goliath-120b-exl2 (there's a different branch for each size) Some of them I've had to do myself eg. I wanted a Q2 GGUF of Falcon 180b There's a guy on huggingface called "TheBloke" who does GGUF, AWQ and GPTQ for most models. For exl2, you can usually just search for exl2 and find them.
- ComputerGuru 3y agoThanks, friend!