For this specific implementation here's info from llama.cpp repo: Perplexity - model options 5.5985 - 13B, q4_0 5.9565 - 7B, f16 6.3001 - 7B, q4_1 6.5949 -

For this specific implementation here's info from llama.cpp repo: Perplexity - model options 5.5985 - 13B, q4_0 5.9565 - 7B, f16 6.3001 - 7B, q4_1 6.5949 - 7B, q4_0 6.5995 - 7B, q4_0, --memory_f16 According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2]. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa [2] https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323