2 ms·
Which quantization are you running and what context size? 32tok/s for that model on that card sounds pretty good to me!
by mongrelion 7mo ago
Which quantization are you running and what context size? 32tok/s for that model on that card sounds pretty good to me!