6 ms·

Can someone ELI5 "LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights"?

by elil17 4y ago

MacsHeadroom 4y ago
LoRA finds a subset of the original weights (about 1%) which can be trained to achieve about the same result as training the whole model while using 100x less compute. Original weights frozen = Rather than modify the original model, the training results are saved to a small file of only a few MB. In practice this means you can fine tune a 30B parameter model on a consumer GPU in a couple of hours. Without LoRA you would need to run multiple expensive data center GPUs for days or weeks.
- stephanheijl 4y ago
  To be more exact, LoRA adds two matrices `A` and `B` to any layers that contain trainable weights. The original weights (`W_0`) have the shape `d × k`. These are frozen. Matrix `A` has dimensions `d × <rank>` (`rank` is configurable) and matrix `B` has the shape `<rank> × k`. A and B are then multiplied and added to `W_0` to get altered weights. The benefit here is that the extra matrices are small compared to `W_0`, which means less parameters need to be optimized, so less activations need to be stored in memory.
  - grph123dot 4y ago
    Your explanation is crystal clear. I suppose it works well in practice, but is there any reason it works that well?
    - stu2b50 4y ago
      Per the original paper, empirically it’s been found that neural network weights often have low intrinsic rank. It follows, then, that the change in the weights as you train also have low intrinsic rank, which means that you should be able represent them with a lower rank matrix.
      - grph123dot 4y ago
        Since we are in ELI5, it seems that the concept of low rank approximation is required to understand this method. (1) https://en.wikipedia.org/wiki/Low-rank_approximation https://en.wikipedia.org/wiki/Low-rank_approximation Edited: By the way, it seems to me that there is an error in the wikipedia page because if the Low-rank approximation takes a larger rank then the bound of the error should decrease, and in this page the error increases.
      - grph123dot 4y ago
        >> that the change in the weights as you train also have low intrinsic rank It seems that the initial matrix of weights has a low rank approximation A and this implies that the difference E = W - A is small, also it seems that PCA fails when E is sparse because PCA is designed to be optimum when the error is gaussian.
        stu2b50 4y ago
        In terms of PCA, PCA is also quite expensive computationally. Additionally, you'd probably have to do SVD instead. Since the weights are derived from gradient descent, yeah we don't really know what the distributions would be. A random projection empirically works quite well for very high dimensions, and is of course very cheap computationally.
      - seydor 4y ago
        Does this mean the matrices are highly compressible?
        loxias 4y ago
        kinda/yes. To translate to more intuitive concepts: the matrices don't contain much variance in as many degrees of freedom as they could. Think of a point cloud of a piece of paper floating in the wind. It would be a 3xn list of points, but "really" it's a 2d piece of paper. Just like I can rewrite the number 27 as 333 or 8+19 or (2^3)+(2^4)+3.. Given a single matrix one can find myriad ways to rewrite it as a sequence of matrices that have the same (or similar) numeric value, but with interesting or desirable properties. :D My favorite example (which is used in signal processing) is to take your ugly matrix and rewrite it as a set of smaller matrices where most of the elements are zero, or a power of 2. It turns out, computers can multiply by zeros and powers of two very fast
  - twic 4y ago
    Ah, so the resulting model contains both the large matrix of original weights, and also the two small matrices of alterations? But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. Why is fine-tuning done with separate alterations, rather than by mutating the original weights?
    - stu2b50 4y ago
      > But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. It's actually larger. If you just have two equally large matrices of the same dimension, one original, and one of "altercations"... then you can just add them together. > Why is fine-tuning done with separate alterations, rather than by mutating the original weights? Then you'd have to compute the gradients for the whole network, which is very expensive when the model has 7b, 65b, 165b parameters. The intent is to make that cheaper by only computing gradients for a low rank representation of the change in the weight matrix from training.
      - gliptic 4y ago
        Correct me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
        stu2b50 4y ago
        I mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
        jprafael 4y ago
        Computing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.
  - seydor 4y ago
    Can rank decomposition be used to reduce the original weight matrices as well? Or are they assumed to be compressed already?
    - metanonsense 4y ago
      Those fully trained networks are usually considered full-rank. At least that is what they say in the LoRA paper.
- quest88 4y ago
  Is this the same as Knowledge Distillation (teacher-student training)?
- tylerekahn 4y ago
  It’s actually as low as 0.01% of the original weights. From the LoRa paper: >When the pre-trained model is GPT-3 175B, the number of train- able parameters |Θ| can be as small as 0.01% of |Φ0|.
- pffft8888 4y ago
  Is this the same as or similar to the Lottery Ticket concept from a few years ago?
- arugulum 4y ago
  >In practice this means you can fine tune a 30B parameter model on a consumer GPU in a couple of hours. Consumer GPU, yes, but in practice LoRA doesn't actually reduce training time. What it mainly reduces is memory requirements. In fact LoRA training can often require more training steps than full fine-tuning and therefore be slower (you can imagine why this is the case: the optimization is trying to modify the mode's behavior a smaller number of parameters, and so has a harder job)
  - MacsHeadroom 4y ago
    Modern peft methods with LoRA actually do reduce training time by orders of magnitude. Here's an example of 20 seconds per epoch on a single consumer GPU: https://github.com/johnsmith0031/alpaca_lora_4bit/issues/7#issue-1635228539 https://github.com/johnsmith0031/alpaca_lora_4bit/issues/7#i...
- esperent 4y ago
  > about the same result Can you qualify this? Is it still useful or not?
  - MacsHeadroom 4y ago
    >Is it still useful or not? Is what still useful? A LoRA is about as good and useful as a full fine tune. If you have unlimited storage space to store them or unlimited compute to make them then I would still prefer full fine tunes. But the difference is marginal and generally not worth the storage space or increased compute costs for individuals.
edwardjhu 4y ago
Hi! I'm the author of the repo. The insight is that we don't need to modify a lot of parameters to get a generally competent model to do well on specific tasks. When you have a linear layer with a weight matrix of dimension d_in x d_out, the change you undergo during full finetuning is also a matrix of d_in x d_out, which can be huge. We represent the latter using two matrices of shape d_in x r and r x d_out. You save a lot of parameters when r is small. So when you use it, the input goes through two streams 1) the orignal frozen weight turning a vector of size d_in to d_out and 2) the low-rank weights turning a vector of size d_in to r and r to d_out. The two streams are then summed together. (There's a figure in the paper.) This way of doing thing is nice for a few reasons. It's easy to parallelize. You can change r to control how many parameters to train. You can also merge the low-rank weights with the original one to avoid latency. Note that we don't select a subset of the original parameters. We train extra ones.
- eternalban 4y ago
  Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.) I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model? And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?
  - edwardjhu 4y ago
    This paper tries something like that https://arxiv.org/pdf/2202.13914.pdf https://arxiv.org/pdf/2202.13914.pdf The gain isn't that significant. We don't understand what these low-rank updates represent, and they might not correspond to "skills" that humans have.
- loxias 4y ago
  Hi! I in _no way_ mean to detract or malign or "anything negative" the parent comment (communication is hard!!), BUT I really compliment that exact sentence. :) My background contains signal processing, "pre-deep learning ML", systems engineering, and firmware, and that sentence jumped out at me as crystal clear in my mind, despite not knowing what HuggingFace is or PyTorch. Correct me if I'm wrong: These huge models involve lots of weights used in large matrices. The contribution of this work is to plug in some matrix factorization and learn a lower dimensional representation, instead of a large second matrix. Fantastic! Also makes me wonder what other performance improvements await through proper application of established and well known Mathematics. :D

Can someone ELI5 "LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights"?

by elil17 4y ago

Can someone ELI5 "LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights"?

MacsHeadroom 4y ago

LoRA finds a subset of the original weights (about 1%) which can be trained to achieve about the same result as training the whole model while using 100x less compute. Original weights frozen = Rather than modify the original model, the training results are saved to a small file of only a few MB. In practice this means you can fine tune a 30B parameter model on a consumer GPU in a couple of hours. Without LoRA you would need to run multiple expensive data center GPUs for days or weeks.

stephanheijl 4y ago
To be more exact, LoRA adds two matrices `A` and `B` to any layers that contain trainable weights. The original weights (`W_0`) have the shape `d × k`. These are frozen. Matrix `A` has dimensions `d × <rank>` (`rank` is configurable) and matrix `B` has the shape `<rank> × k`. A and B are then multiplied and added to `W_0` to get altered weights. The benefit here is that the extra matrices are small compared to `W_0`, which means less parameters need to be optimized, so less activations need to be stored in memory.
- grph123dot 4y ago
  Your explanation is crystal clear. I suppose it works well in practice, but is there any reason it works that well?
  - stu2b50 4y ago
    Per the original paper, empirically it’s been found that neural network weights often have low intrinsic rank. It follows, then, that the change in the weights as you train also have low intrinsic rank, which means that you should be able represent them with a lower rank matrix.
    - grph123dot 4y ago
      Since we are in ELI5, it seems that the concept of low rank approximation is required to understand this method. (1) https://en.wikipedia.org/wiki/Low-rank_approximation https://en.wikipedia.org/wiki/Low-rank_approximation Edited: By the way, it seems to me that there is an error in the wikipedia page because if the Low-rank approximation takes a larger rank then the bound of the error should decrease, and in this page the error increases.
    - grph123dot 4y ago
      >> that the change in the weights as you train also have low intrinsic rank It seems that the initial matrix of weights has a low rank approximation A and this implies that the difference E = W - A is small, also it seems that PCA fails when E is sparse because PCA is designed to be optimum when the error is gaussian.
      - stu2b50 4y ago
        In terms of PCA, PCA is also quite expensive computationally. Additionally, you'd probably have to do SVD instead. Since the weights are derived from gradient descent, yeah we don't really know what the distributions would be. A random projection empirically works quite well for very high dimensions, and is of course very cheap computationally.
    - seydor 4y ago
      Does this mean the matrices are highly compressible?
      - loxias 4y ago
        kinda/yes. To translate to more intuitive concepts: the matrices don't contain much variance in as many degrees of freedom as they could. Think of a point cloud of a piece of paper floating in the wind. It would be a 3xn list of points, but "really" it's a 2d piece of paper. Just like I can rewrite the number 27 as 333 or 8+19 or (2^3)+(2^4)+3.. Given a single matrix one can find myriad ways to rewrite it as a sequence of matrices that have the same (or similar) numeric value, but with interesting or desirable properties. :D My favorite example (which is used in signal processing) is to take your ugly matrix and rewrite it as a set of smaller matrices where most of the elements are zero, or a power of 2. It turns out, computers can multiply by zeros and powers of two very fast
- twic 4y ago
  Ah, so the resulting model contains both the large matrix of original weights, and also the two small matrices of alterations? But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. Why is fine-tuning done with separate alterations, rather than by mutating the original weights?
  - stu2b50 4y ago
    > But this is smaller than the alternative of a model which contains the large matrix of original weights, and an equally large matrix of alterations. It's actually larger. If you just have two equally large matrices of the same dimension, one original, and one of "altercations"... then you can just add them together. > Why is fine-tuning done with separate alterations, rather than by mutating the original weights? Then you'd have to compute the gradients for the whole network, which is very expensive when the model has 7b, 65b, 165b parameters. The intent is to make that cheaper by only computing gradients for a low rank representation of the change in the weight matrix from training.
    - gliptic 4y ago
      Correct me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
      - stu2b50 4y ago
        I mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
        jprafael 4y ago
        Computing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.
- seydor 4y ago
  Can rank decomposition be used to reduce the original weight matrices as well? Or are they assumed to be compressed already?
  - metanonsense 4y ago
    Those fully trained networks are usually considered full-rank. At least that is what they say in the LoRA paper.
quest88 4y ago
Is this the same as Knowledge Distillation (teacher-student training)?
tylerekahn 4y ago
It’s actually as low as 0.01% of the original weights. From the LoRa paper: >When the pre-trained model is GPT-3 175B, the number of train- able parameters |Θ| can be as small as 0.01% of |Φ0|.
pffft8888 4y ago
Is this the same as or similar to the Lottery Ticket concept from a few years ago?
arugulum 4y ago
>In practice this means you can fine tune a 30B parameter model on a consumer GPU in a couple of hours. Consumer GPU, yes, but in practice LoRA doesn't actually reduce training time. What it mainly reduces is memory requirements. In fact LoRA training can often require more training steps than full fine-tuning and therefore be slower (you can imagine why this is the case: the optimization is trying to modify the mode's behavior a smaller number of parameters, and so has a harder job)
- MacsHeadroom 4y ago
  Modern peft methods with LoRA actually do reduce training time by orders of magnitude. Here's an example of 20 seconds per epoch on a single consumer GPU: https://github.com/johnsmith0031/alpaca_lora_4bit/issues/7#issue-1635228539 https://github.com/johnsmith0031/alpaca_lora_4bit/issues/7#i...
esperent 4y ago
> about the same result Can you qualify this? Is it still useful or not?
- MacsHeadroom 4y ago
  >Is it still useful or not? Is what still useful? A LoRA is about as good and useful as a full fine tune. If you have unlimited storage space to store them or unlimited compute to make them then I would still prefer full fine tunes. But the difference is marginal and generally not worth the storage space or increased compute costs for individuals.

edwardjhu 4y ago

Hi! I'm the author of the repo. The insight is that we don't need to modify a lot of parameters to get a generally competent model to do well on specific tasks. When you have a linear layer with a weight matrix of dimension d_in x d_out, the change you undergo during full finetuning is also a matrix of d_in x d_out, which can be huge. We represent the latter using two matrices of shape d_in x r and r x d_out. You save a lot of parameters when r is small. So when you use it, the input goes through two streams 1) the orignal frozen weight turning a vector of size d_in to d_out and 2) the low-rank weights turning a vector of size d_in to r and r to d_out. The two streams are then summed together. (There's a figure in the paper.) This way of doing thing is nice for a few reasons. It's easy to parallelize. You can change r to control how many parameters to train. You can also merge the low-rank weights with the original one to avoid latency. Note that we don't select a subset of the original parameters. We train extra ones.

eternalban 4y ago
Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.) I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model? And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?
- edwardjhu 4y ago
  This paper tries something like that https://arxiv.org/pdf/2202.13914.pdf https://arxiv.org/pdf/2202.13914.pdf The gain isn't that significant. We don't understand what these low-rank updates represent, and they might not correspond to "skills" that humans have.
loxias 4y ago
Hi! I in _no way_ mean to detract or malign or "anything negative" the parent comment (communication is hard!!), BUT I really compliment that exact sentence. :) My background contains signal processing, "pre-deep learning ML", systems engineering, and firmware, and that sentence jumped out at me as crystal clear in my mind, despite not knowing what HuggingFace is or PyTorch. Correct me if I'm wrong: These huge models involve lots of weights used in large matrices. The contribution of this work is to plug in some matrix factorization and learn a lower dimensional representation, instead of a large second matrix. Fantastic! Also makes me wonder what other performance improvements await through proper application of established and well known Mathematics. :D