2 ms·
Can you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
by syntaxing 8d ago
Can you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
- naasking 8d agoModel, Q4_K_M: https://huggingface.co/agentionai/Signal-3.8-27B-GGUF https://huggingface.co/agentionai/Signal-3.8-27B-GGUF DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan. Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
- syntaxing 8d agoThanks! Have you seen issues with quantizing the kv cache?