4 ms·
? All the research I've seen says quantization has basically negligible performance impact. My experience working with 65B 4bit has been great
by Taek 4y ago
?
All the research I've seen says quantization has basically negligible performance impact. My experience working with 65B 4bit has been great
- syntaxing 4y agoI agree with OP. Only the nonquantized models has given me good results too. I only have used the 7B and 13B. I don't have enough computing power to run 65B.