3 ms·
Aye was on quantized weights using gptq.
by lumost 3y ago
Aye was on quantized weights using gptq.
- LoganDark 3y agoTry GGML, llama.cpp is pretty fast
- lumost 3y agomakes sense - I ultimately need to train the weights so was focusing on GPTQ, I'll try out ggml and see if the latency is better. I have some flexibility on whether I run inference/training on the same model instance. What context length were you using? I was maxing ~2048 tokens, which may also explain the apparent latency.
- LoganDark 3y agollama.cpp builds a prefix cache so the only latency is on the first generation :)