I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other. As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass. With dense Qwen 27B on 3090/llama.cpp I get: - no MTP: 1x42, 2x33, 3x24, 4x19 t/s - MTP: 1x50, 2x30, 3x33, 4x30 t/s