2 ms·
> But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. It's act
by stu2b50 4y ago
> But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations.
It's actually larger. If you just have two equally large matrices of the same dimension, one original, and one of "altercations"... then you can just add them together.
> Why is fine-tuning done with separate alterations, rather than by mutating the original weights?
Then you'd have to compute the gradients for the whole network, which is very expensive when the model has 7b, 65b, 165b parameters. The intent is to make that cheaper by only computing gradients for a low rank representation of the change in the weight matrix from training.
- gliptic 4y agoCorrect me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
- stu2b50 4y agoI mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
- jprafael 4y agoComputing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.
- arugulum 4y ago>Then you'd have to compute the gradients for the whole network You have to do that with LoRA regardless, to compute the gradients for the lowest-level LoRA weights.