LoRA is an alternative to traditional fine tuning (which is usually done on specific layers as you mentioned). To quote the LoRA paper[1]: > We hypothesize th

LoRA is an alternative to traditional fine tuning (which is usually done on specific layers as you mentioned). To quote the LoRA paper[1]: > We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen It's truly revolutionary: It basically lets you create a very small "diff" which you apply yo an existing model and it is suddenly fine tuned. These diff models are very small (5M for example). [1] https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685