3 ms·

as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?

cabidaher 2y ago
This paper [1] does atempt that and reports similar performance compared to conventional pre-training. However, they do start off by doing a normal full-rank training and claim that it is needed to 'warm start' the training process. [1] https://arxiv.org/abs/2307.05695 https://arxiv.org/abs/2307.05695
- danielhanchen 2y ago
  Oh yes this paper! The main issue is the scaling of the A and B LoRA matrices. Some papers show scaling the B matrix with larger learning rates (LoRA+) could be beneficial. DoRA for eg learns an auto scaling vector of numbers which tries to alleviate these issues. Galore might be more equivalent to full pretraining with the gradients being low rank.
sp332 2y ago
Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?
- buildbot 2y ago
  I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
  - cheald 2y ago
    It's worse than that, because lora requires two matrices per layer. At full rank, you have an additional NxN parameters to learn versus full finetuning, where N is min(input_features, output_features). For example, tuning a layer of 128 in x 256 out is 32k params. Learning a full-rank lora for that layer would be two matrices of 128x128 and 128x256 = 48k params.
    - buildbot 2y ago
      Yeah, exactly. Though the 48k param lora might be as good as a 48k param layer of higher rank, I haven't looked into that case really.
buildbot 2y ago
Yes, I’ve tested this out. It does train, but the scaling doesn’t seem to pan out. It’ll perform slightly better than the number of trainable parameters, but never improves as you scale, so for now there’s no benefit.
whimsicalism 2y ago
i would be shocked if this worked well

cabidaher 2y ago

This paper [1] does atempt that and reports similar performance compared to conventional pre-training. However, they do start off by doing a normal full-rank training and claim that it is needed to 'warm start' the training process. [1] https://arxiv.org/abs/2307.05695 https://arxiv.org/abs/2307.05695

danielhanchen 2y ago
Oh yes this paper! The main issue is the scaling of the A and B LoRA matrices. Some papers show scaling the B matrix with larger learning rates (LoRA+) could be beneficial. DoRA for eg learns an auto scaling vector of numbers which tries to alleviate these issues. Galore might be more equivalent to full pretraining with the gradients being low rank.

sp332 2y ago

Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?

buildbot 2y ago
I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
- cheald 2y ago
  It's worse than that, because lora requires two matrices per layer. At full rank, you have an additional NxN parameters to learn versus full finetuning, where N is min(input_features, output_features). For example, tuning a layer of 128 in x 256 out is 32k params. Learning a full-rank lora for that layer would be two matrices of 128x128 and 128x256 = 48k params.
  - buildbot 2y ago
    Yeah, exactly. Though the 48k param lora might be as good as a 48k param layer of higher rank, I haven't looked into that case really.

buildbot 2y ago

Yes, I’ve tested this out. It does train, but the scaling doesn’t seem to pan out. It’ll perform slightly better than the number of trainable parameters, but never improves as you scale, so for now there’s no benefit.

whimsicalism 2y ago

i would be shocked if this worked well