3 ms·
I fixed some chat template issues for llama.cpp and other inference engines! To run it, do: ./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -
by danielhanchen 1y ago
I fixed some chat template issues for llama.cpp and other inference engines! To run it, do:
./llama.cpp/llama-cli -hf unsloth/SmolLM3-3B-GGUF:Q4_K_XL --jinja -ngl 99
- segmondy 1y agodoing the good work, thanks daniel!
- danielhanchen 1y agoThank you!
- v5v3 1y agoThanks
- danielhanchen 1y agoThanks!
- diggan 1y ago> fixed some chat template issues This seems to be a persistent issue with almost all weight releases, even from bigger companies like Meta. Are the people who release these weights not testing them in various inference engines? Seems they make it work with Huggingface's Transformers library, then call it a day, but sometimes not even that.
- clarionbell 1y agoNo they don't. Why would they? Most of them are using a single inference engine, most likely developed inhouse. Or they go for something like vLLM, but llama.cpp especially is under their radar. The reason is simple. There isn't much money in it. llama.cpp is free and targets lower end of the hardware spectrum. Corporations will run something else, or even more likely, offload the task to contractor.
- danielhanchen 1y agoThe chat template issues are actually not on llama.cpp's side, but on all engines (including vLLM, SGLang etc) For eg see https://www.reddit.com/r/unsloth/comments/1l97eaz/deepseekr10528_updated_with_many_fixes_especially/ https://www.reddit.com/r/unsloth/comments/1l97eaz/deepseekr1... - which fixed tool calling for DeepSeek R1
- danielhanchen 1y agoOh so chat template issues yes are quite pervasive sadly - for eg Llama as you mentioned, but also Qwen, Mistral, Google, the Phi team, DeepSeek - it's actually very common! My take is large labs with closed source models also did have issues during the beginning, but most likely have standardized the chat template (for eg OpenAI using ChatML). The OSS community on the other hand keeps experimenting with new templates - for example adding tool calling causes a large headache. For example in https://unsloth.ai/blog/phi3 https://unsloth.ai/blog/phi3 - we found many bugs in OSS models.