3 ms·

Just run /goal to optimise it and you should be good in less than an hour. Also best to use models that support speculative decoding.

LoganDark 2mo ago
Optimize llama.cpp? Hmm. WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry. Fable recommended n-gram speculation so I'm working on that now.
- mirekrusin 2mo ago
  Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins. ps. ngram didn't work for me very well, but dedicated speculative model works very well ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like: # over baseline at temp 0.6 (95 vs 45 tok/s), ~4x over naive layer-split baseline. Qwen3.6-27B-MTP-UD-Q8_K_XL: ./llama.cpp/llama-server \ -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q8_K_XL \ --no-mmproj \ --parallel 1 \ --kv-unified \ --flash-attn on \ --fit off \ --split-mode tensor \ -ngl 99 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --host 0.0.0.0 \ --tools all \ --jinja \ --ctx-size 262144 \ --spec-type draft-mtp \ --spec-draft-n-max 6 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 1.1 \ --threads 8 \ --reasoning-budget 2048 \ --reasoning on \ --chat-template-kwargs '{"preserve_thinking": true}' \ --reasoning-budget-message "reasoning budget consumed, time to answer now" ... Qwen: Qwen3.6 Qwen3.6: Qwen3.6-27B Qwen3.6-35B-A3B: Qwen3.6-35B-A3B-MTP Qwen3.6-35B-A3B-MTP: Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL Qwen3.6-27B: Qwen3.6-27B-MTP Qwen3.6-27B-MTP: Qwen3.6-27B-MTP-UD-Q8_K_XL
  - LoganDark 2mo ago
    I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.
    - mirekrusin 2mo ago
      Yeah for that dense models are not great better to pick some MoE to get benefits of large vram pool while keeping reasonable speed.

LoganDark 2mo ago

Optimize llama.cpp? Hmm. WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry. Fable recommended n-gram speculation so I'm working on that now.

mirekrusin 2mo ago
Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins. ps. ngram didn't work for me very well, but dedicated speculative model works very well ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like: # over baseline at temp 0.6 (95 vs 45 tok/s), ~4x over naive layer-split baseline. Qwen3.6-27B-MTP-UD-Q8_K_XL: ./llama.cpp/llama-server \ -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q8_K_XL \ --no-mmproj \ --parallel 1 \ --kv-unified \ --flash-attn on \ --fit off \ --split-mode tensor \ -ngl 99 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --host 0.0.0.0 \ --tools all \ --jinja \ --ctx-size 262144 \ --spec-type draft-mtp \ --spec-draft-n-max 6 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 1.1 \ --threads 8 \ --reasoning-budget 2048 \ --reasoning on \ --chat-template-kwargs '{"preserve_thinking": true}' \ --reasoning-budget-message "reasoning budget consumed, time to answer now" ... Qwen: Qwen3.6 Qwen3.6: Qwen3.6-27B Qwen3.6-35B-A3B: Qwen3.6-35B-A3B-MTP Qwen3.6-35B-A3B-MTP: Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL Qwen3.6-27B: Qwen3.6-27B-MTP Qwen3.6-27B-MTP: Qwen3.6-27B-MTP-UD-Q8_K_XL
- LoganDark 2mo ago
  I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.
  - mirekrusin 2mo ago
    Yeah for that dense models are not great better to pick some MoE to get benefits of large vram pool while keeping reasonable speed.