3 ms·
Hey HN family! I found a few bugs for Phi-4 - Microsoft's latest MIT licensed LLM to be on par with GPT-4o mini 1. End of sentence should be <|im_end|> not <|e
by danielhanchen 2y ago
Hey HN family! I found a few bugs for Phi-4 - Microsoft's latest MIT licensed LLM to be on par with GPT-4o mini
1. End of sentence should be <|im_end|> not <|endoftext|>
2. Chat template should not auto add an assistant prompt
3. Padding token should not be EOS but <|dummy_87|>
I also converted Phi-4 to Llama-arch. I uploaded GGUFs, 4bit quants, dynamic quants and all fixes to https://huggingface.co/unsloth https://huggingface.co/unsloth
I also made a Colab notebook to finetune Phi-4 on a free GPU: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb https://colab.research.google.com/github/unslothai/notebooks...
- sunaookami 2y agoWasn't Phi-3 also bugged/is still bugged? Seems like Microsoft just doesn't care. >to be on par with GPT-4o mini Phi is known to overfit benchmarks. It's way, way worse then that.
- throwaway314155 2y agoAnecdotally, I've been experimenting with Phi-4 the past hour or so (so, yeah, not very comprehensive) and it's certainly a strong model. Definitely better than the previous Phi models.
- danielhanchen 2y agoYep Phi-4 definitely is better than Phi-3.5!
- danielhanchen 2y agoPhi-3 should be fixed as well - but yes there were bugs as well! https://x.com/danielhanchen/status/1782853167572832650 https://x.com/danielhanchen/status/1782853167572832650 Phi-3's sliding window should be 2048 and not 2047, and they also had chat template issues - I uploaded correct versions to https://huggingface.co/unsloth/Phi-3.5-mini-instruct https://huggingface.co/unsloth/Phi-3.5-mini-instruct
- simonw 2y agoHuh! That may explain why I kept on getting visible <|im_end|> output when I tried running a Phi-4 GGUF file using llama.cpp.
- danielhanchen 2y agoOh yes exactly! I trimmed it out now :) The better chat template should be: {% for message in messages %}{% if (message['role'] == 'system') %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'assistant') %}{{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}
- CGamesPlay 2y ago> We converted Phi-4 to Llama’s architecture for better accuracy and easier use. What does this mean? When I think about "model architecture", I think about the number of weights in each layer, the organization of the layers, etc. And AFAIK, it's untenable to "port" a model from one to the other without effectively retraining it. So what does it actually mean to "convert to Llama's architecture"?
- Sn0wCoder 2y agoWould guess GGUF so you can run on llama.cpp, LM Studio, etc..., but OP can hopefully clarity further for you.
- danielhanchen 2y agoYep converting to Llama arch definitely makes accessibility much better - also many fast LLM serving libraries normally support Llama, so it makes it easier to port and use!
- danielhanchen 2y agoOh Phi-4's architecture is inspired from Llama itself, except they merged the attention matrices into 1 large matrix for better FLOP utilization, and the gate/up matrices in the MLP. Phi-3 use to use sliding window attention, but they got rid of that in Phi-4. So, you can "Mistral-fy" Phi-3 and convert it to Mistral arch (by unmerging the merges), and now you can "Llama-fy" Phi-4 to Llama arch. The reason why accuracy increases in finetuning is because during LoRA finetuning, you learn only 1 A matrix for merged QKV, whilst unmerging it creates 3 A matrices - this allows the model to have more freedom to learn new features.
- behnamoh 2y agoI know some of those words... Man, do you recommend any blog/book/etc. that teaches me how to know this stuff? Most books are either too low level or too high level.
- 2y ago
- sroussey 2y agoCan you convert to ONNX so I can try in web browser?
- sroussey 2y agoWould like to update this: https://huggingface.co/spaces/webml-community/phi-3.5-webgpu https://huggingface.co/spaces/webml-community/phi-3.5-webgpu
- danielhanchen 2y agoOh I can probs try doing this!
- mugivarra69 2y ago[dead]
- magicalhippo 2y ago> I uploaded GGUFs, 4bit quants, dynamic quants The dynamic quantization[1] looks really interesting. Now, I've just been dabbling, but did I understand correctly that this dynamic quantization is compatible with GUFF? If so, how do you convert it? Just the standard way or? I was really curious to try the dynamic 4-bit version of the Llama-3.2 11B Vision model as I found the Q8 variant much better than the standard Q4_K_M variant in certain cases, but it doesn't fully fit my GPU so is significantly slower. [1]: https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit
- danielhanchen 2y agoOh the dynamic 4bit quants sadly are not GGUF compatible yet - it currently works through Hugging Face transformers, Unsloth and other trainers. My goal was to make a dynamic quant for GGUF as well - it's just a tad bit complicated to select which layers to quantize and not with GGUF - I might have to manually edit the llama.cpp quantize C file Also I'm unsure yet if llama.cpp as of 11th Jan 2025 supports Llama Vision (or maybe it's new?) I do remember Qwen / Llava type models are working
- magicalhippo 2y ago> Oh the dynamic 4bit quants sadly are not GGUF compatible yet Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? > Also I'm unsure yet if llama.cpp as of 11th Jan 2025 supports Llama Vision (or maybe it's new?) Ah, I totally forgot Ollama did that on their own and didn't merge upstream. I'm using Ollama because was so easy to get running on my main Windows rig so I can take advantage of my GPU there, I still do a bit of gaming, while all the stuff which uses Ollama for inference I run on my server. Anyway thanks for the response.
- danielhanchen 2y ago> Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue? Just code! Technically I was working on dynamic quants for DeepSeek V3 (200GB in size), which will increase accuracy by a lot for a 2bit model (if you leave attention in 4bit), and just use 20GB more. -> But I'm still working on it! > Ah, I totally forgot Ollama did that on their own and didn't merge upstream. Yep they have Llama vision support! Llama.cpp has Qwen, Llava support - I think Llama V support is coming, but it'll take much more time - the arch is vastly different than normal transformers due to cross attention