4 ms·
Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was ve
by kristianp 16d ago
Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.
- Pragmata 15d agoI got a RTX5080 as primary and a RTX3050 (8gb) as secondary. I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that. With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)