5 ms·
LoRA+: Efficient Low Rank Adaptation of Large Models
- yau8edq12i 2y agoWhat an unfortunate name... I initially thought this was about wireless communication. https://en.wikipedia.org/wiki/LoRa https://en.wikipedia.org/wiki/LoRa
- bee_rider 2y agoThe idea of low rank approximations is not new, truncated SVDs for example have been used for many decades.
- yau8edq12i 2y agoThe acronym LoRA used in the context of deep learning (2021) is about seven years younger than the radio communication protocol called LoRa (2014). Type "lora" in a search engine and see what you get.
- kleiba 2y agoIn the first 10 search results, I now get a mix of results for either of the two technologies when searching with Google.
- blamestross 2y agoYeah I would prefer they didn't offusicate the actually useful search term
- sorenjan 2y agoThis gets mentioned here everytime an article about LoRA is posted. Sometimes acronyms means multiple things, they're not in the same field so the risk of confusion beyond short headlines is negligible. It's a bit like if someone reading a bicycling article and getting annoyed that FTP means Functional Threshold Power instead of File Transfer Protocol, or reading about machine learning and getting confused that MLP doesn't mean My Little Pony.
- rakoo 2y ago"computer science" and "bicycles" aren't the same domain, it's fine to have the same acronym. "computer science" and "tv shows" aren't the same domain, it's fine to have the same acronym. "computer science" and "computer science" are the same domain, it's not a good idea to use the same acronym.
- WithinReason 2y agoLarge Models is in the title so it's obviously not about radio
- GuB-42 2y agoThe acronym is also spelled out in the title: LoRA = Low Rank Adaptation.
- squigz 2y agoContext is hard!
- rakoo 2y agoSo instead of LoRa and anything else, everyone now has to say LoRa (the communication protocol) or LoRa (the large model thing). Having to add context all the time makes everything so much simpler !
- WithinReason 2y agoLow rank adaptation is abbreviated LoRA
- squigz 2y agoOr potentially include the necessary context in the title of the post.
- rakoo 2y ago
- rytill 2y agoThat’s LoRa. This is LoRA.
- kcorbitt 2y agoThis specific variant "LoRA+" described in this paper is even harder to search for. I was doing some research on this technique recently and it turns out that "Lora+" matches with "Lora" in Discord search, which is quite unhelpful. :)
- SquareWheel 2y agoDiscord search is one of the worst I've ever used. They remap words like "localization" to "local", which makes it impossible to search for more specific terms.
- cuuupid 2y agoI’m struggling to understand from this paper whether the approach is better in the general sense (all cases, with wider models seeing greater benefits) or purely for wider models (with narrower models seeing detriment)? If it’s the former this could effectively halve finetuning cost overnight which would go a significant way towards enabling a wider array of use cases for LoRA.
- batterseapower 2y agoThe other recent improvement suggested for LoRA is DoRA: https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch https://magazine.sebastianraschka.com/p/lora-and-dora-from-s.... It really does seem to strongly outperform LoRA - see also https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.htm...
- WithinReason 2y agoThe two methods seem to be independent, wonder if you can combine them for even better performance. Interestingly both seem to indirectly modify the optimisation process, in my opinion effectively trying to fix a bad optimiser. Seems like we still have a long way to go after Adam...
- neodypsis 2y ago> Seems like we still have a long way to go after Adam... A preprint in arxiv suggests that Adam works better than SGD for training LLMs due to the issue of class-imbalance [0]. It appears that scaling the gradient step helps with the training, for example, see another approach suggested in [1]. 0. https://arxiv.org/pdf/2402.19449 https://arxiv.org/pdf/2402.19449 1. https://arxiv.org/pdf/2402.02347 https://arxiv.org/pdf/2402.02347
- josalhor 2y agoI just skimmed over LoRA+ and DoRA and I see no reason why these improvements could not go hand in hand. Actually, LoRA+ seems to be about efficient training while DoRA seems about improving the ability to actually learn, making it significantly more robust. Although I still have my questions on how the improvements of LoRA+ would be applied to the magnitude vector.
- Ger_Onimo 2y agoI've just started playing with DoRAs for fine-tuning TTS models towards particular styles of speech, and they're working extremely well!
- axpy906 2y agoIn 2024 are folks still swapping out LoRA adapters? Is this still relevant?
- youssefabdelm 2y agoA better name would've probably been FastLoRA or something
- throwaway2562 2y agofLORA
- mobilemidget 2y agoI was expecting to read about LOng RAng radio communication. https://en.wikipedia.org/wiki/LoRa https://en.wikipedia.org/wiki/LoRa
- ironbound 2y agoI've had sucess with GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection https://arxiv.org/abs/2403.03507 https://arxiv.org/abs/2403.03507
- Scipio_Afri 2y agoThis uses less memory so you can do fine tuning or hardware with less vram but at a cost of taking longer on training - there is a throughput penalty, the paper detailing the technique shows something like a 15% decrease in throughput.
- allpaca 2y agoThis is old, having been released in February... Why do you talk about it now?