Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at high

Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating. [0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...