I'd have to try the KV cache trick but folks get pretty competitive speeds with the current 31B/27B dense models e.g. https://www.reddit.com/r/LocalLLaMA/commen

I'd have to try the KV cache trick but folks get pretty competitive speeds with the current 31B/27B dense models e.g. https://www.reddit.com/r/LocalLLaMA/comments/1tc9j6u/mi50s_qwen_36_27b_528_tps_tg_1569_tps_pp_no_mtp/ https://www.reddit.com/r/LocalLLaMA/comments/1tc9j6u/mi50s_q...