3 ms·
Quantizing it to 8-bit basically eliminates its ability to write code.
by zamalek 4y ago
Quantizing it to 8-bit basically eliminates its ability to write code.
- Taek 4y ago? All the research I've seen says quantization has basically negligible performance impact. My experience working with 65B 4bit has been great
- syntaxing 4y agoI agree with OP. Only the nonquantized models has given me good results too. I only have used the 7B and 13B. I don't have enough computing power to run 65B.