Also I would highly recommend Q5_K_M for both 7B and 13B models. It has the best balance between quality and weight of the model and almost indistinguishable f

Also I would highly recommend Q5_K_M for both 7B and 13B models. It has the best balance between quality and weight of the model and almost indistinguishable from original f16: https://www.reddit.com/r/LocalLLaMA/comments/142q5k5/updated_relative_comparison_of_ggml_quantization/ https://www.reddit.com/r/LocalLLaMA/comments/142q5k5/updated...