4 ms·
Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separ
by zozbot234 6mo ago
Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separate drafting model) and then verify them all at once. It's an emerging feature still (not widely supported) and it's only useful for speeding up highly predictable token runs, but it's one way to do better in practice than the common-sense theoretical limit might suggest.
- pbgcp2026 6mo agoIt seems to me it's only Grok 4.20 that does this currently? Which other models did you have in mind, if I may ask?
- phamilton 6mo agoGemma4, qwen3.6, deepseek v4, mimo, glm 5/5.1 all do MTP.
- pbgcp2026 6mo agoThank you, I just realised we are talking about MTP. It seems that it's not that clear though. "Currently, the MTP capabilities are primarily accessible through Google's proprietary LiteRT framework, rather than the open-weights versions... Despite the missing MTP heads in the open release, Gemma 4 (specifically the 26B-A4B variant) still demonstrates high efficiency"
- parsimo2010 6mo agoIf Mistral Medium 3.5 supports it, that might get it to 10 t/s. It will still be fairly slow.
- freakynit 6mo ago[dead]