12 ms·
LoRA: Low-Rank Adaptation of Large Language Models
- pgen 4y agoName clash! https://en.wikipedia.org/wiki/LoRa#LoRaWAN https://en.wikipedia.org/wiki/LoRa#LoRaWAN
- mronetwo 4y agoit's Microsoft. they know. they just don't care
- ga_to 4y agoMicrosoft has done this before with mauikit and mauilinux: https://github.com/dotnet/maui/issues/35 https://github.com/dotnet/maui/issues/35 Unlikely that they even consider checking whether they are stomping across existing names.
- capableweb 4y ago> Unlikely that they even consider checking whether they are stomping across existing names. Or it's on purpose as existing terms already have good amount of search traffic for those terms, and Microsoft know Google/Bing will rank Microsoft's own pages higher than what's already out there.
- Semaphor 4y agoThey even do that with their own products…
- capableweb 4y agoEasy, one is LoRA and the other one is LoRa, Microsoft made it very distinct, as they always do.
- fullstop 4y agoJust don't put the files in the same directory on an exFAT drive
- capableweb 4y agoor on macOS, although I'm not sure it's still by default using case sensitive file system or not. I do remember the first time that bit me though, being a programmer using Linux collaborating with a developer using macOS. Must have been in ~2005 or something.
- fullstop 4y agoThe one which bit me happened when I was running a java minimizer / obfuscator on a Windows platform and it assumed that A.class was not the same as a.class. It worked great on Linux and didn't warn that it had overwritten a file, resulting in a package which almost worked.
- dudeinjapan 4y agoAnd yet they sued Mike Rowe who made Mike Rowe Soft.
- deleted 4y ago[deleted]
- smusamashah 4y agoThis term is already used when fine tuning stable diffusion models. https://replicate.com/blog/lora-faster-fine-tuning-of-stable-diffusion https://replicate.com/blog/lora-faster-fine-tuning-of-stable...
- pygy_ 4y agoYour case insensitive brain doesn't get it... It's LoRA, not LoRa. /s
- eternalban 4y agoTIL learned about LoRA via https://news.ycombinator.com/item?id=35287740 https://news.ycombinator.com/item?id=35287740 See also: Huggingface PEFT: https://github.com/huggingface/peft https://github.com/huggingface/peft
- nummerfuenf 4y agoCan we stop naming things like stuff that already exists?
- postdb 4y agoFirebird!!!
- wlesieutre 4y agoThis is totally different, Microsoft's A is capitalized! https://lora-alliance.org/ https://lora-alliance.org/
- entropicdrifter 4y agoUnfortunately, no. It's even worse within the video game industry. I'm not just talking Doom 4, er, Doom (2016). The upcoming sequel to 2014's Lords of The Fallen? Well that's called Lords of The Fallen. They didn't even get 2 game in before repeating the exact same name.
- Agentlien 4y agoMy favourite video game franchise in terms of confusing name is the Jedi Knight franchise. Star Wars: Dark Forces Star Wars Jedi Knight: Dark Forces 2 Star Wars Jedi Knight 2: Jedi Outcast Star Wars Jedi Knight: Jedi Academy
- indeyets 4y agohttps://github.com/microsoft/LoRA/issues/47 https://github.com/microsoft/LoRA/issues/47
- denysvitali 4y agoI was looking for this comment. Thank you!
- runnerup 4y agoThere’s a not insignificant intersection of projects and developers who might be using both LoRA and LoRa at the same time. What a terrible name collision. Hopefully this doesn’t become one of the foundational terms in AI that everyone must use frequently like “Transformer”.
- asddubs 4y agoyeah it really does seem like the AI folks are EEEing EE
- Maxion 4y agoI was going to comment the same, horrible name collision. Surprised they didn't notice it.
- indeyets 4y agohttps://github.com/microsoft/LoRA/issues/47 https://github.com/microsoft/LoRA/issues/47
- seydor 4y agotransformer itself is ambiguous But to be clear, LoRa is not related to ANN training, is it? Why would they be using both?
- whalesalad 4y agotransformer and adapter are two of the new "ai terms" that grind my gears
- ahkurtz 4y agoIsn't there something really perfect about people working on a language model either not trying or outright failing to use that language model to tell them if their project name already exists? On their github they reference a related project called "HuggingFace" so you know the sky's the limit with the names in this field, could have been called anything else really.
- chaorace 4y ago
- krossitalk 4y agoMaybe call it LoRALLMR (Laura Loomer)
- muny 4y agoWhy use the same name as LoRa? https://lora-alliance.org/ https://lora-alliance.org/ Edit: Microsoft is even a member of the LoRa alliance: https://lora-alliance.org/lora-alliance-press-release/microsoft-joins-the-lora-alliance-board-of-directors/ https://lora-alliance.org/lora-alliance-press-release/micros...
- anthk 4y agoAlso Guix vs Guix...
- elcomet 4y agoThere are many more things called lora. https://en.m.wikipedia.org/wiki/Lora https://en.m.wikipedia.org/wiki/Lora It doesn't really matter as long as it's not in the same field. No one will be confused between the two.
- magicalhippo 4y ago> No one will be confused between the two. Except search engines...
- Filligree 4y agoThat's okay, Bing-GPT doesn't get confused.
- FlyingRobot 4y agoI had to scan the readme to make sure this story wasn’t about applying machine learning to radio communication.
- ChancyChance 4y agoSmall CNNs can be used for BLE channel hopping and body detection.
- brodouevencode 4y agohttps://en.wikipedia.org/wiki/LoRa https://en.wikipedia.org/wiki/LoRa for the communications architecture
- elil17 4y agoCan someone ELI5 "LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights"?
- MacsHeadroom 4y agoLoRA finds a subset of the original weights (about 1%) which can be trained to achieve about the same result as training the whole model while using 100x less compute. Original weights frozen = Rather than modify the original model, the training results are saved to a small file of only a few MB. In practice this means you can fine tune a 30B parameter model on a consumer GPU in a couple of hours. Without LoRA you would need to run multiple expensive data center GPUs for days or weeks.
- stephanheijl 4y agoTo be more exact, LoRA adds two matrices `A` and `B` to any layers that contain trainable weights. The original weights (`W_0`) have the shape `d × k`. These are frozen. Matrix `A` has dimensions `d × <rank>` (`rank` is configurable) and matrix `B` has the shape `<rank> × k`. A and B are then multiplied and added to `W_0` to get altered weights. The benefit here is that the extra matrices are small compared to `W_0`, which means less parameters need to be optimized, so less activations need to be stored in memory.
- grph123dot 4y agoYour explanation is crystal clear. I suppose it works well in practice, but is there any reason it works that well?
- stu2b50 4y agoPer the original paper, empirically it’s been found that neural network weights often have low intrinsic rank. It follows, then, that the change in the weights as you train also have low intrinsic rank, which means that you should be able represent them with a lower rank matrix.
- Der_Einzige 4y agoI really hope this doesn't displace regular fine tuning techniques. Dreambooth is superior in quality to Lora with image generation, and I suspect that it's similar with LLMs.
- eternalban 4y agohttps://dreambooth.github.io/ https://dreambooth.github.io/ The LoRA paper’s ‘problem statement’ makes a compelling case for practical benefits of the approach. Specifically, no added latency, no serial processing bottlenecks, shared baseline model, compact time/space requirements. How does dreambooth stack up in this regard?
- mattnewton 4y agoIn the image space, dreambooth full-model tunes can handle multiple concepts and tend to be easier to get hard/complex things like a person’s likeness correct. I’ve found that LoRA tunes struggle to be accepted by people as producing their own face compared to full dreambooth models tuned on the same inputs, most likely because we are very sensitive to facial differences of faces we are very familiar with. I haven’t seen this effect for styles or other kinds of concepts, where people are a little less sensitive about the fidelity. LoRA is much easier to train, easier to compose, and can have the base model swapped out in many cases though so if it’s good enough for the concept you are trying to add to the model it’s often worth the subtle quality loss.
- stu2b50 4y agoI suspect it’s not that similar. The intuition behind LoRA is more true the higher the rank of the weights of the model. Even the smallest LLMs have considerably higher rank weights than Stable Diffusion. They are large, after all.
- brucethemoose2 4y agoThere are some WIP evolutions of SD Lora in the works, like locon and lycoris. https://github.com/KohakuBlueleaf/LyCORIS https://github.com/KohakuBlueleaf/LyCORIS
- eternalban 4y agoFrom the paper: “Aghajanyan et al. (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace.” Would be great to have an informed practitioner comment (sota) on why we opt for random projection. Is the actual ‘intrinsic’ vector space uncomputable? Too slow to find?
- moyix 4y agoNot an informed/sota practitioner, but isn't this just a standard property of high dimensional spaces? https://en.wikipedia.org/wiki/Random_projection https://en.wikipedia.org/wiki/Random_projection > The core idea behind random projection is given in the Johnson-Lindenstrauss lemma, which states that if points in a vector space are of sufficiently high dimension, then they may be projected into a suitable lower-dimensional space in a way which approximately preserves the distances between the points.
- stu2b50 4y agoRandom projects work well in high dimensional spaces, they’re cheap, easy, and require no understanding of the initial space. Part of the point of Lora is efficiency, after all!
- loxias 4y ago> Is the actual ‘intrinsic’ vector space uncomputable? Too slow to find? OMG yes. It's also an unsolved problem, once can approximate it. I've written several non-parametric blind arbitrary dimension DR algorithms and been obsessed with the space most of my life. If you think O(n^2) feels slow, try O(n^3)...:) For more, read about mean shift clustering, or the new hot stuff: topological data analysis/bar codes/mapper algorithm.
- timmg 4y agoThis sounds similar to "prompt tuning": https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html https://ai.googleblog.com/2022/02/guiding-frozen-language-mo...
- stu2b50 4y agoIt’s actually completely different. What you linked is about zero shot learning by adjusting the prompt, vs Lora which is about actually fine tuning the weights of the model.
- timmg 4y agoIn that case, you can think of the prompt as being one vector of the model that is being tuned while the rest is frozen. Not exactly the same, to be sure. But fulfills a similar need: more efficient "fine tuning" of a large model.
- stu2b50 4y agoI suppose that is true. You can even train the prompt with gradient descent. But in practice, it ends up being fairly different.
- eternalban 4y agoThey address prompt tuning's issues in the paper: "The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods." https://ar5iv.labs.arxiv.org/html/2106.09685 https://ar5iv.labs.arxiv.org/html/2106.09685 This is key imo: "More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task".
- numlocked 4y agoFor those wondering why this is interesting: This technique is being used to reproduce[0] the Alpaca results from Stanford[1] with a few hours of training on consumer-grade hardware. I believe there will soon be a cottage industry of providing application-specific fine-tuned models like this, that can run in e.g. AWS very inexpensively. The barrier today seems to be that the base model (here, Meta's LLaMA) is encumbered and can't be used commercially. Someone will soon, I'm confident, release e.g. an MIT-licensed equivalent and we'll all be off to the races. [0] https://github.com/tloen/alpaca-lora https://github.com/tloen/alpaca-lora [1] https://crfm.stanford.edu/2023/03/13/alpaca.html https://crfm.stanford.edu/2023/03/13/alpaca.html
- outside1234 4y agoOr, more importantly than in AWS, locally in disconnected or poorly connected scenarios like in-vehicle or in-home.
- romanzubenko 4y agoToday Databricks announced [0] 6b parameter model from EleutherAI finetuned on Alpaca dataset. According to their CEO[1], training took 3 hours, and costed $30. They didn't release any details on how it was trained, but likely with LoRa. [0] https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html https://www.databricks.com/blog/2023/03/24/hello-dolly-democ... [1] https://twitter.com/alighodsi/status/1639251347777388544 https://twitter.com/alighodsi/status/1639251347777388544
- numlocked 4y agoInteresting. I wonder what the training cost was for: https://huggingface.co/EleutherAI/gpt-neox-20b https://huggingface.co/EleutherAI/gpt-neox-20b Perhaps it’s in the paper…
- michaelhartm 4y agoThey used the 6b GPT4-J, not 20B. That's what's interesting, it's a smallish large language model :).
- htrp 4y agothis is from 2 years ago
- sharemywin 4y agoCould the Bloom model be used with this training to build a commercial allowed small-ish model?
- stu2b50 4y agoIt cheapens the cost of fine tuning, it doesn’t make the model itself smaller at inference time.
- outside1234 4y agoHas anyone diaried out a good learning path for going from a larger pre-trained model to a fine tuned model? Trying to understand all of the parts here but it sort of hard to fine anything linear...
- rdedev 4y agoCame across this library in the past where you can easily add LoRA and other efficient fine tuning techniques easily into huggingface models. Haven't tried it though and support for different models may be limited https://adapterhub.ml/ https://adapterhub.ml/
- leobg 4y agoI’m wondering what the difference is between LoRA and adapters. Especially I’m wondering whether LoRA is being used for LLaMA because it is proven to be better or simply because it happened to be the first such technique that worked and now everyone just uses it. In other words: Is it worth it trying the adapter approach with models like LLaMA?
- TheFaheem 4y agoThe difference is that this inserts adapter layers on top of the model. In contrast, LoRA decomposes the model weight matrices using low-rank decomposition. So, LoRA increases finetuning performance by reducing parameter numbers whereas Adapters increases efficiency by keeping the pretrained model frozen (and only tunes a small number of parameters added to the model).
- JustSomeNobody 4y agoAssholes. Don't call it LoRA! There's already a technology called LoRa! Fuck I hate this crap. Be better than this.
- krolden 4y agoIts Microsoft they'll only be better if they go under.
- alecco 4y ago*17 Jun 2021
- jprafael 4y agoIf this works, is there any theory why training models with low rank layers (y = (A.B).x + b) directly doesnt work? (or do they?)
- zeckalpha 4y agoQuite different from https://en.m.wikipedia.org/wiki/LoRa https://en.m.wikipedia.org/wiki/LoRa
- michaelhartm 4y agoBtw, it's kinda crazy how bad the GPT4-J results in the blog are compared to the Dolly one, which seem pretty good. Do we know why it works so well to use this 50k dataset?
- quadrature 4y agoDolly is instruction fine tuned whereas GPT4-J is not. Which means that it doesn't even understand that it is being instructed to do something, it is just doing an autocomplete.
- wruza 4y agoIf you read this thread and still wonder what the heck and why “fine-tuning” is, here’s a good starter explanation I found: https://www.baeldung.com/cs/fine-tuning-nn https://www.baeldung.com/cs/fine-tuning-nn