4 ms·
> LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimate
by mountainriver 1y ago
> LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimated in terms of dataset size
I’m shocked they didn’t look at progressive merging of LoRAs. Research shows that’s the best way of improving its ability to model higher level features.
Seems like a massive miss, not to mention there is other research that contradicts a lot of their findings. This feels a bit like a researchers first pass at learning LoRA
- yenepho 1y agoI am curious, would you mind sharing a citation?
- Mkengin 1y agohttps://arxiv.org/abs/2311.13600 https://arxiv.org/abs/2311.13600 https://arxiv.org/abs/2410.22911 https://arxiv.org/abs/2410.22911 https://arxiv.org/abs/2409.16167 https://arxiv.org/abs/2409.16167
- mountainriver 1y agoDon’t forget ReLoRA! https://arxiv.org/abs/2307.05695 https://arxiv.org/abs/2307.05695
- let_tim_cook_ 1y agoI'm not sure why progressive LoRa merging needs to be addressed here. They show there is a regime of problem where LoRa performs equivalently to FFT. Progressive merging of LoRa is somewhere inbetween and categorically more complex than just LoRa so would be dominated by standard LoRa in that case. While progressive merging could train faster as fewer params are trainable at any given time, it results in very larger adapter diffs OTO the size of the original model and doesn't retain the benefits of being able to deploy multiple adapters over the same base model idt.