2 ms·

I've been using it for the past few days, and it runs really well! I usually get 7 token/s using llama or lm studio, but this inference recipe runs at a smooth

I've been using it for the past few days, and it runs really well! I usually get 7 token/s using llama or lm studio, but this inference recipe runs at a smooth 80 tokens per second. Genuinely very usable, and fully local!

kristianp 16d ago
Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.
- Pragmata 16d ago
  I got a RTX5080 as primary and a RTX3050 (8gb) as secondary. I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that. With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)
falsaberN1 16d ago
With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all)
- Pragmata 16d ago
  it was but i max out on context so it doesn't all fit with kv cache etc...
blagui 16d ago
7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.
- supermatt 16d ago
  > 7 t/s not sure how you can use this 80 t/s - maybe you need to finish reading a sentence before you comment on it?
davada 14d ago
Btw, using llama.cpp you can achieve 55 - 45 token/second for processing/generation. if you use a qwen3.8-27B (IQ4_XS) verison, with decent quality in reasoning for coding/tool usage (with a 16GB nvidia). I think now most of the struggle is getting a 32B-27B llm to work on 16GB/12GB card, because they are at least affordable/accessible for the time being compared to higher end models. Recently the nvidia RTX 5090 32GB has reached price range of 7500 dollars (despite MSRP being around 2000 dollars when it was first launched). Crazy times.

kristianp 16d ago

Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.

Pragmata 16d ago
I got a RTX5080 as primary and a RTX3050 (8gb) as secondary. I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that. With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)

falsaberN1 16d ago

With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all)

Pragmata 16d ago
it was but i max out on context so it doesn't all fit with kv cache etc...

blagui 16d ago

7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.

supermatt 16d ago
> 7 t/s not sure how you can use this 80 t/s - maybe you need to finish reading a sentence before you comment on it?

davada 14d ago

Btw, using llama.cpp you can achieve 55 - 45 token/second for processing/generation. if you use a qwen3.8-27B (IQ4_XS) verison, with decent quality in reasoning for coding/tool usage (with a 16GB nvidia). I think now most of the struggle is getting a 32B-27B llm to work on 16GB/12GB card, because they are at least affordable/accessible for the time being compared to higher end models. Recently the nvidia RTX 5090 32GB has reached price range of 7500 dollars (despite MSRP being around 2000 dollars when it was first launched). Crazy times.