3 ms·

> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does

> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks. > --spec-draft-n-max 5 5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.

hypfer 2mo ago
Yes to both. The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though. As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute. Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
- hedgehog 2mo ago
  Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.
  - hypfer 2mo ago
    I wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions. Though it might turn out that that doesn't offer any benefits. Has it been tried yet? ___ EDIT: LLM pointed me towards this thing I'm not going to read https://arxiv.org/abs/2512.11280 https://arxiv.org/abs/2512.11280 "AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference" Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
    - ndriscoll 2mo ago
      Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
      - hedgehog 2mo ago
        This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
    - petu 2mo ago
      Deepseek's DSpark does dynamically adjust speculated token count per user/completion. https://arxiv.org/abs/2607.05147 https://arxiv.org/abs/2607.05147 But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.
freehorse 2mo ago
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
hypfer 2mo ago
Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33 The unweighted average is 4.1 So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5. Also, cool that it carries over from qwen3.6 like that.

hypfer 2mo ago

Yes to both. The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though. As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute. Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.

hedgehog 2mo ago
Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.
- hypfer 2mo ago
  I wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions. Though it might turn out that that doesn't offer any benefits. Has it been tried yet? ___ EDIT: LLM pointed me towards this thing I'm not going to read https://arxiv.org/abs/2512.11280 https://arxiv.org/abs/2512.11280 "AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference" Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
  - ndriscoll 2mo ago
    Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
    - hedgehog 2mo ago
      This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
  - petu 2mo ago
    Deepseek's DSpark does dynamically adjust speculated token count per user/completion. https://arxiv.org/abs/2607.05147 https://arxiv.org/abs/2607.05147 But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.

freehorse 2mo ago

I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.

hypfer 2mo ago

Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33 The unweighted average is 4.1 So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5. Also, cool that it carries over from qwen3.6 like that.