4 ms·

If i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HM

by seydor 4y ago

If i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa45zITSg.png https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa4... train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong

PoignardAzur 4y ago
I had to read the paper first, but yeah, that diagram is shockingly simple once you get it. Some annotations: - The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero". - d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.) - r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.
xkgt 4y ago
Correct me if I am wrong, to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.
- pavo-etc 4y ago
  Yes you still require the original model weights to use LoRA layers. For many LLaMA based models you need to find the original weight yourself and then apply the LoRA diff on top of that.
- seydor 4y ago
  > to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.

by seydor 4y ago

PoignardAzur 4y ago

I had to read the paper first, but yeah, that diagram is shockingly simple once you get it. Some annotations: - The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero". - d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.) - r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.

xkgt 4y ago

Correct me if I am wrong, to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.

pavo-etc 4y ago
Yes you still require the original model weights to use LoRA layers. For many LLaMA based models you need to find the original weight yourself and then apply the LoRA diff on top of that.
seydor 4y ago
> to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right? You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.