Koboldcpp [1], which builds on llamacpp and adds a gui, is a great way to run these models. Most people aren't running these models at full weight, ggml quantiz

Koboldcpp [1], which builds on llamacpp and adds a gui, is a great way to run these models. Most people aren't running these models at full weight, ggml quantization is recommended for cpu+gpu or gptq if you have the gpu vram. GGML 13b models at 4bit (Q4_0) take somewhere around 9gb of ram and q5_K_M take about 11gb. Gpu offloading support has also been added, I've been using 22 layers on my laptop rtx 2070 max q 8gb vram and CLBlast. I get around ~2-3 tokens per second with 13b models. In my experience, running 13b models is worth the extra time it takes to generate a response compared to 7b models. GPTQ is faster, I think, but I can't fit a quantized 13b model so I don't use it. TheBloke [2] has been quantizing models and uploading them to HF and will probably upload a quantized version of this online soon. His discord server also has good guides to help you get going, linked in the model card of most of his models. https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp https://huggingface.co/TheBloke https://huggingface.co/TheBloke Edit: There's a bug with the newest Nvidia drivers that causes speed slowdown with large context size. I downgraded and stayed on 531.61 . The theory is that newer drivers change how out of cuda memory management works when trying to avoid OOM errors. https://www.reddit.com/r/LocalLLaMA/comments/1461d1c/major_performance_degradation_with_nvidia_driver/ https://www.reddit.com/r/LocalLLaMA/comments/1461d1c/major_p... https://github.com/vladmandic/automatic/discussions/1285 https://github.com/vladmandic/automatic/discussions/1285