4 ms·
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your worklo
by naasking 8d ago
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
- syntaxing 8d agoCan you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
- naasking 8d agoModel, Q4_K_M: https://huggingface.co/agentionai/Signal-3.8-27B-GGUF https://huggingface.co/agentionai/Signal-3.8-27B-GGUF DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan. Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
- syntaxing 8d agoThanks! Have you seen issues with quantizing the kv cache?