Nice but you are leaving some performance on the table (if you have a GPU) Exllama + GPTQ is the way to go llama.cpp && GGUF are great on CPUs More data: ht

Nice but you are leaving some performance on the table (if you have a GPU) Exllama + GPTQ is the way to go llama.cpp && GGUF are great on CPUs More data: https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacp...