3 ms·
MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 https://github.com/ggml-org/llama.cpp/pu
by zdw 5mo ago
MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 https://github.com/ggml-org/llama.cpp/pull/20533) and I'd imagine Gemma 4 will come soon.
The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
- dakolli 5mo agoyet, still mostly useless.
- EGreg 5mo agoHow does this get added in practice?
- flakiness 5mo agoAccording to the linked PR, the original model does come with MTP which is another "head" (=output path) in the same model and (supposedly) runs faster. The current implementation ignores that head but the PR let the tool recognize it, plus does proper integration (run the MTP while running the slower main path then compare the result, I believe.)
- flebron 5mo agoThe standard way of doing MTP is to run the drafter autoregressively for k steps, and then (not concurrently) use the larger model as a verifier for those k tokens at the same time. The larger model can then accept a prefix of those k tokens, and in any case generates one more token (which is needed in case you accepted zero tokens from the drafter). The larger model can effectively use this k as a "batch" dimension, reducing the penalty of large weight loading. Meanwhile the drafter is much smaller, so it's fine for _it_ to be autoregressive, as long as the main model is parallel.
- WhitneyLand 5mo agoYeah important conceptually to remember MTP is kind of just more weights, but speculative decoding is the runtime algorithm that’s a significant add to whatever code is serving the model.
- HumanOstrich 5mo agoThat is.. inaccurate.
- WhitneyLand 5mo agoHow so? I’m not saying most of work doesn’t go into creating the drafting model or enabling a new head on the primary model, but the point is that however cool it is the result is, more weights. Speculative decoding requires code to be aware of how this works at the inference level.
- tarruda 5mo agoThere is a newer PR which will probably be merged soon: https://github.com/ggml-org/llama.cpp/pull/22673 https://github.com/ggml-org/llama.cpp/pull/22673
- entropicdrifter 5mo agoOllama merged a PR for MTP about 2 hours ago, as well: https://github.com/ollama/ollama/pull/15980 https://github.com/ollama/ollama/pull/15980 Edit: Seems they also have a pre-release version out with the functionality added: https://github.com/ollama/ollama/releases/tag/v0.23.1-rc0 https://github.com/ollama/ollama/releases/tag/v0.23.1-rc0
- theturtle32 5mo agoSad: theturtle32@ai1:~$ ollama run gemma4:31b-coding-mtp-bf16 pulling manifest Error: pull model manifest: 412: this model requires macOS
- zozbot234 5mo agoWhat's "sad" is how slow the ollama folks are being in vendoring newer versions of ggml into their codebase. That attitude just leaves them stranded without access to newer features.
- deleted 5mo ago[deleted]
- xlayn 5mo agoOhhhh geee!!! I just applied the patch to my local git copy. You need to use the model on the PR that he submitted, the model is particular because it has extra information that allows the MTP to happen. I have two amd gpus, and qwen3.6 27B qk6 does around 20t/s generation... If I run it only on one I get like 35t/s. But with this patch I saw 46t/s with qwen3.6 27B q8... this is insane, it's 250% faster than the original speed, there was no gpu I could upgrade to get that kind of boost, amazing!
- basch 5mo agoI have a dumb performance question. Why when asking a model to change text in a minor way; are we not asking it to generate the operational transformations necessary to modify the text, and then just executing the ot on the existing text vs reproducing every token? Maybe tools are doing that more than I realize?
- XYen0n 5mo agoThe only thing a model can output is tokens; to achieve this, a tool of converting tokens into operational transformations is required. For example, I have an ast-grep skill, it will instruct the model to generate ast-grep rules and run ast-grep to perform file modifications.
- basch 5mo agoI am saying to directly output the operational transformation instructions as the tokens. You’re essentially telling it to “write the diff” and then applying the patch. [retain(8), delete(6), insert("very very"), retain(10)]
- ritonlajoie 5mo agothere is a model in openrouter doing exactly this, it generates diffs. forgot the name though
- mike_hearn 5mo agoOpenAI models emit a format similar to a regular diff, but without the line numbers. Look at apply_patch
- cryptoz 5mo agoThis is the approach I take with code edits to existing files at Code+=AI; I wrote a blog post with a simple example of AST modification to illustrate: https://codeplusequalsai.com/static/blog/prompting_llms_to_modify_existing_code_using_asts.html https://codeplusequalsai.com/static/blog/prompting_llms_to_m...
- 5mo ago
- nzeid 5mo agoA few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.
- apexalpha 5mo agoI’ve been swapping between these too as well. However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.
- sigmoid10 5mo agoGemma certainly was trained for tool calling, but the implementation in llama.cpp has been troubled because Gemma uses a different chat template format. The processor from the transformers library works fine though.
- apexalpha 5mo agoOh I must've missed this. The AI space moves so fast! I'll check it out again.
- intothemild 5mo agoDon't forget to update the gguf you have too. The templates in them were updated recently too
- nzeid 5mo agoI'm using llama.cpp with Gemma and tool calling is mission critical. It's perfectly fine on my end. There are definitely differences in the eagerness to tool-call that you'll need to manage. And for all local models I've ever used, I've had to micromanage the tools provided by servers to eliminate any possibility that they reach for something wonky or confusing.
- magicalhippo 5mo ago
- fridder 5mo agoI'd love to see this in oMLX too. It has been a rather nice tool
- egeres 5mo agoThere's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978 https://github.com/ggml-org/llama.cpp/issues/21978, I can't wait to see how it will compare against MTP
- endymi0n 5mo agoI don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?
- neonstatic 5mo agoMTP allows for a smaller draft model to supply tokens to the larger model for verification. If tokens are good enough, the larger model can accept them instead of generating its own, which is much cheaper. From what I read, this is not unique to GGUF or MLX format. Instead, the model has to be trained to support that feature.
- nullc 5mo agoThanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.
- _factor 5mo agoTook 2x AMD MI50s to 50 t/s instead of 20 t/s for Q8 27B. Impressive.