4 ms·
Have you (or others!) done much research on using different subsets of target modules? Just doing q, k, v, o is dramatically less ram and much more accessible (
by jasonjmcghee 3y ago
Have you (or others!) done much research on using different subsets of target modules? Just doing q, k, v, o is dramatically less ram and much more accessible (mistral 7b on 3080ti)
- tarruda 3y agoI haven't, but didn't write the article. I don't understand what these modules mean for fine-tuning Here someone does Alpaca LoRA fine-tuning on Llama 7b using only q/v modules: https://colab.research.google.com/drive/1rqWABmz2ZfolJOdoy6TRc6YI7d128cQO?usp=sharing https://colab.research.google.com/drive/1rqWABmz2ZfolJOdoy6T...
- jasonjmcghee 3y agoThey are different parts of the network. If you don't fine tune them, they don't change. They are "frozen"- if you're familiar with fine tuning neural networks
- tarruda 3y agoHow does one decide which of these modules shouls be fine tuned for different tasks?