3 ms·

I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for

I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.

naasking 8d ago
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model. Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
- syntaxing 8d ago
  Can you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
  - naasking 8d ago
    Model, Q4_K_M: https://huggingface.co/agentionai/Signal-3.8-27B-GGUF https://huggingface.co/agentionai/Signal-3.8-27B-GGUF DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan. Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
    - syntaxing 8d ago
      Thanks! Have you seen issues with quantizing the kv cache?
UncleOxidant 8d ago
Vulkan or ROCm backend? I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
- syntaxing 8d ago
  Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
  - UncleOxidant 8d ago
    I have not measured pre-fill, but it's said to be around 1000. It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
    - syntaxing 8d ago
      Halogen as in this right? https://github.com/peonist-ai/halogen-flash-server https://github.com/peonist-ai/halogen-flash-server
      - UncleOxidant 8d ago
        yes
        syntaxing 8d ago
        Thanks! I really like how the author packaged everything into a container. Definitely going to give it a go over the weekend!

I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.

naasking 8d ago

I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model. Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.

syntaxing 8d ago
Can you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
- naasking 8d ago
  Model, Q4_K_M: https://huggingface.co/agentionai/Signal-3.8-27B-GGUF https://huggingface.co/agentionai/Signal-3.8-27B-GGUF DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan. Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
  - syntaxing 8d ago
    Thanks! Have you seen issues with quantizing the kv cache?

UncleOxidant 8d ago

Vulkan or ROCm backend? I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.

syntaxing 8d ago
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
- UncleOxidant 8d ago
  I have not measured pre-fill, but it's said to be around 1000. It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
  - syntaxing 8d ago
    Halogen as in this right? https://github.com/peonist-ai/halogen-flash-server https://github.com/peonist-ai/halogen-flash-server
    - UncleOxidant 8d ago
      yes
      - syntaxing 8d ago
        Thanks! I really like how the author packaged everything into a container. Definitely going to give it a go over the weekend!