3 ms·

The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization sho

The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization should be very small.

xdennis 4y ago
What's the 4chan version?
- aseipp 4y ago
  See https://github.com/ggerganov/llama.cpp/issues/62 https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)
- cypress66 4y ago
  https://rentry.org/llama-tard-v2 https://rentry.org/llama-tard-v2