3 ms·
What do you use to run them? My 3060 is generating just 40token per second with llama.cpp
by avereveard 3y ago
What do you use to run them? My 3060 is generating just 40token per second with llama.cpp
- eyegor 3y agoI use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.