4 ms·
> We converted Phi-4 to Llama’s architecture for better accuracy and easier use. What does this mean? When I think about "model architecture", I think about th
by CGamesPlay 2y ago
> We converted Phi-4 to Llama’s architecture for better accuracy and easier use.
What does this mean? When I think about "model architecture", I think about the number of weights in each layer, the organization of the layers, etc. And AFAIK, it's untenable to "port" a model from one to the other without effectively retraining it. So what does it actually mean to "convert to Llama's architecture"?
- Sn0wCoder 2y agoWould guess GGUF so you can run on llama.cpp, LM Studio, etc..., but OP can hopefully clarity further for you.
- danielhanchen 2y agoYep converting to Llama arch definitely makes accessibility much better - also many fast LLM serving libraries normally support Llama, so it makes it easier to port and use!
- danielhanchen 2y agoOh Phi-4's architecture is inspired from Llama itself, except they merged the attention matrices into 1 large matrix for better FLOP utilization, and the gate/up matrices in the MLP. Phi-3 use to use sliding window attention, but they got rid of that in Phi-4. So, you can "Mistral-fy" Phi-3 and convert it to Mistral arch (by unmerging the merges), and now you can "Llama-fy" Phi-4 to Llama arch. The reason why accuracy increases in finetuning is because during LoRA finetuning, you learn only 1 A matrix for merged QKV, whilst unmerging it creates 3 A matrices - this allows the model to have more freedom to learn new features.
- behnamoh 2y agoI know some of those words... Man, do you recommend any blog/book/etc. that teaches me how to know this stuff? Most books are either too low level or too high level.
- danielhanchen 2y agoIf it helps there a few YouTube recordings of conferences and workshops I did about stuff! Low level Technicals of LLMs: https://www.youtube.com/watch?v=pRM_P6UfdIc https://www.youtube.com/watch?v=pRM_P6UfdIc CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA https://www.youtube.com/watch?v=hfb_AIhDYnA Chat with PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0 https://www.youtube.com/watch?v=MQwryfkydc0 PyTorch Conference talk here: https://www.youtube.com/watch?v=PdtKkc5jB4g https://www.youtube.com/watch?v=PdtKkc5jB4g