3 ms·
Oh Phi-4's architecture is inspired from Llama itself, except they merged the attention matrices into 1 large matrix for better FLOP utilization, and the gate/u
by danielhanchen 2y ago
Oh Phi-4's architecture is inspired from Llama itself, except they merged the attention matrices into 1 large matrix for better FLOP utilization, and the gate/up matrices in the MLP.
Phi-3 use to use sliding window attention, but they got rid of that in Phi-4.
So, you can "Mistral-fy" Phi-3 and convert it to Mistral arch (by unmerging the merges), and now you can "Llama-fy" Phi-4 to Llama arch.
The reason why accuracy increases in finetuning is because during LoRA finetuning, you learn only 1 A matrix for merged QKV, whilst unmerging it creates 3 A matrices - this allows the model to have more freedom to learn new features.
- behnamoh 2y agoI know some of those words... Man, do you recommend any blog/book/etc. that teaches me how to know this stuff? Most books are either too low level or too high level.
- danielhanchen 2y agoIf it helps there a few YouTube recordings of conferences and workshops I did about stuff! Low level Technicals of LLMs: https://www.youtube.com/watch?v=pRM_P6UfdIc https://www.youtube.com/watch?v=pRM_P6UfdIc CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA https://www.youtube.com/watch?v=hfb_AIhDYnA Chat with PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0 https://www.youtube.com/watch?v=MQwryfkydc0 PyTorch Conference talk here: https://www.youtube.com/watch?v=PdtKkc5jB4g https://www.youtube.com/watch?v=PdtKkc5jB4g