4 ms·
When working with GGUF what chat templates do you use? Pretty much every gguf I've imported into ollama has given me garbage response. Converting the tokenize
by vergessenmir 2y ago
When working with GGUF what chat templates do you use? Pretty much every gguf I've imported into ollama has given me garbage response. Converting the tokenizer json has yielded mixed results.
For example how do you handle the phi-4 models gguf chat template?
- simonw 2y agoI use whatever what template is baked into the GGUF file. You can click on the little info icon on Hugging Face to see that directly. For https://huggingface.co/matteogeniaccio/phi-4/tree/main?show_file_info=phi-4-Q4_K_M.gguf https://huggingface.co/matteogeniaccio/phi-4/tree/main?show_... that's this: {% for message in messages %}{% if (message['role'] == 'system') %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|><|im_start|>assistant<|im_sep|>'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}