4 ms·
> Would love to hear folks inference setups, the A100 was... not fast - but I didn't spend any time trying to make it fast. What do you mean? I get something l
by LoganDark 3y ago
> Would love to hear folks inference setups, the A100 was... not fast - but I didn't spend any time trying to make it fast.
What do you mean? I get something like 25 tokens per second on an RTX 3060 12G. Try using quantized weights, the full-size ones are only for training.
- lumost 3y agoAye was on quantized weights using gptq.
- LoganDark 3y agoTry GGML, llama.cpp is pretty fast
- lumost 3y agomakes sense - I ultimately need to train the weights so was focusing on GPTQ, I'll try out ggml and see if the latency is better. I have some flexibility on whether I run inference/training on the same model instance. What context length were you using? I was maxing ~2048 tokens, which may also explain the apparent latency.
- LoganDark 3y agollama.cpp builds a prefix cache so the only latency is on the first generation :)