4 ms·
How do you decide what model variant to use? There's a bunch of Quant method variations of Llama-2-13B-chat-GGML [0], how do you know which one to use? Reading
by TheAceOfHearts 3y ago
How do you decide what model variant to use? There's a bunch of Quant method variations of Llama-2-13B-chat-GGML [0], how do you know which one to use? Reading the "Explanation of the new k-quant methods" is a bit opaque.
[0] https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML
- Charlieholtz 3y agoThis is a great question. My best answer is that there's a speed/intelligence trade-off. The smaller weights (7B) will run faster and require less memory, but you won't get the same quality responses as the 13B / 70B model. I think there may be a Llama 30B variant coming soon too.
- rashidujang 3y agoHey there, I was confused at this exact question too. This link might help, written by a contributor to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 TLDR: Lower quantization means higher perplexity (i.e. how 'confused' the model is when seeing new information). It's a matter of testing it out and choosing a model that fits your available memory. The higher the quantization number, the better (generally).