3 ms·
Did you have to quantize it yourself to 4.75bpw and 3bpw or are they readily available for download?
by ComputerGuru 3y ago
Did you have to quantize it yourself to 4.75bpw and 3bpw or are they readily available for download?
- idonotknowwhy 3y agoMost of the time it's readily available eg: Panchovix/goliath-120b-exl2 (there's a different branch for each size) Some of them I've had to do myself eg. I wanted a Q2 GGUF of Falcon 180b There's a guy on huggingface called "TheBloke" who does GGUF, AWQ and GPTQ for most models. For exl2, you can usually just search for exl2 and find them.
- ComputerGuru 3y agoThanks, friend!