I am happily using qwen2.5-coder-7b-instruct-q3_k_m.gguf with a context size of 32768 on an RTX 3060 Mobile with 6GB VRAM using llama.cpp [2]. With 16GB VRAM, y

I am happily using qwen2.5-coder-7b-instruct-q3_k_m.gguf with a context size of 32768 on an RTX 3060 Mobile with 6GB VRAM using llama.cpp [2]. With 16GB VRAM, you could use qwen2.5-7b-instruct-q8_0.gguf which is basically indistinguishable from the fp16 variant. [1] https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF [2] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp