5 ms·

Fantastic article if you are quick to just go to the comments like I usually do, don't. Read it. One of my favorites: LoRA works by representing model updates

Fantastic article if you are quick to just go to the comments like I usually do, don't. Read it. One of my favorites: LoRA works by representing model updates as low-rank factorizations, which reduces the size of the update matrices by a factor of up to several thousand. This allows model fine-tuning at a fraction of the cost and time. Being able to personalize a language model in a few hours on consumer hardware is a big deal, particularly for aspirations that involve incorporating new and diverse knowledge in near real-time. The fact that this technology exists is underexploited inside Google, even though it directly impacts some of our most ambitious projects. Anyone has worked with LoRa ? Sounds super interesting.

epiccoleman 3y ago
We need to scrape the entire corpus of /r/ASOIAF so it can come up with wild theories about how Tyrion is a secret Targaryen and confirm Benjen == Daario once and for all.
adroitboss 3y ago
You can find the guy who created it on reddit u/edwardjhu. I remember because he showed up in the Stable Diffusion Subreddit. https://www.reddit.com/r/StableDiffusion/comments/1223y27/im_the_creator_of_lora_how_can_i_make_it_better/ https://www.reddit.com/r/StableDiffusion/comments/1223y27/im...
seydor 3y ago
If i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa45zITSg.png https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa4... train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong
- PoignardAzur 3y ago
  I had to read the paper first, but yeah, that diagram is shockingly simple once you get it. Some annotations: - The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero". - d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.) - r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.
- xkgt 3y ago
  Correct me if I am wrong, to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.
  - pavo-etc 3y ago
    Yes you still require the original model weights to use LoRA layers. For many LLaMA based models you need to find the original weight yourself and then apply the LoRA diff on top of that.
  - seydor 3y ago
    > to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.
eulers_secret 3y ago
If you use the web interface (oobabooga), then training a LoRa is as easy as clicking the "training" tab, keeping all the defaults, and giving it a flat text file of your data. The defaults are sane enough to not begin undermining any instruction tuning too much. Takes 3-5 hours on a 3080 for 7B, 4bit model (and ~1KWh). So far I've trained 3: 2 on the entire text of ASOIAF (converted from e-books) and 1 on the Harry Potter series. I can ask questions like "tell me a story about a long winter in Westeros" and get something in the "voice" of GRRM and with real references to the text. It can write HP fanfics all day long. My favorite so far was the assistant self-inserting into a story with Jon Snow, complete with "The Assistant has much data for you. Please wait while it fetches it." and actually having a conversation with Jon. Asking specific questions is way more of a miss (e.x. "Who are Jon Snow's real parents?" returns total BS), but that may be because my 3080 is too weak to train anything other than 7B models in 4bit (which is only supported with hacked patches). I used Koala as my base model. I'm getting close to dropping $1600 on a 4090, but I should find employment first... but then I'll have less time to mess with it.
- hotfixguru 3y ago
  Which model did you use? I read your comment when this HN thread was alive and decided to try to feed it a few websites (such as PEP8[0] for Python), and it only took a few minutes for LoRa training to complete, and the answers were not good. I also have a 3080. [0]: https://peps.python.org/pep-0008/ https://peps.python.org/pep-0008/
- seydor 3y ago
  how much memory does the 7B training need?
  - eulers_secret 3y ago
    ~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
    - 3y ago
Levitz 3y ago
I wholeheartedly second this. This article seems to me to be one important, small piece of text to read. It might very well end up somewhere in a history book someday.

epiccoleman 3y ago

We need to scrape the entire corpus of /r/ASOIAF so it can come up with wild theories about how Tyrion is a secret Targaryen and confirm Benjen == Daario once and for all.

adroitboss 3y ago

You can find the guy who created it on reddit u/edwardjhu. I remember because he showed up in the Stable Diffusion Subreddit. https://www.reddit.com/r/StableDiffusion/comments/1223y27/im_the_creator_of_lora_how_can_i_make_it_better/ https://www.reddit.com/r/StableDiffusion/comments/1223y27/im...

seydor 3y ago

If i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa45zITSg.png https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa4... train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong

PoignardAzur 3y ago
I had to read the paper first, but yeah, that diagram is shockingly simple once you get it. Some annotations: - The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero". - d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.) - r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.
xkgt 3y ago
Correct me if I am wrong, to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.
- pavo-etc 3y ago
  Yes you still require the original model weights to use LoRA layers. For many LLaMA based models you need to find the original weight yourself and then apply the LoRA diff on top of that.
- seydor 3y ago
  > to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.

eulers_secret 3y ago

If you use the web interface (oobabooga), then training a LoRa is as easy as clicking the "training" tab, keeping all the defaults, and giving it a flat text file of your data. The defaults are sane enough to not begin undermining any instruction tuning too much. Takes 3-5 hours on a 3080 for 7B, 4bit model (and ~1KWh). So far I've trained 3: 2 on the entire text of ASOIAF (converted from e-books) and 1 on the Harry Potter series. I can ask questions like "tell me a story about a long winter in Westeros" and get something in the "voice" of GRRM and with real references to the text. It can write HP fanfics all day long. My favorite so far was the assistant self-inserting into a story with Jon Snow, complete with "The Assistant has much data for you. Please wait while it fetches it." and actually having a conversation with Jon. Asking specific questions is way more of a miss (e.x. "Who are Jon Snow's real parents?" returns total BS), but that may be because my 3080 is too weak to train anything other than 7B models in 4bit (which is only supported with hacked patches). I used Koala as my base model. I'm getting close to dropping $1600 on a 4090, but I should find employment first... but then I'll have less time to mess with it.

hotfixguru 3y ago
Which model did you use? I read your comment when this HN thread was alive and decided to try to feed it a few websites (such as PEP8[0] for Python), and it only took a few minutes for LoRa training to complete, and the answers were not good. I also have a 3080. [0]: https://peps.python.org/pep-0008/ https://peps.python.org/pep-0008/
seydor 3y ago
how much memory does the 7B training need?
- eulers_secret 3y ago
  ~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
  - 3y ago

Levitz 3y ago

I wholeheartedly second this. This article seems to me to be one important, small piece of text to read. It might very well end up somewhere in a history book someday.