4 ms·
Came across this library in the past where you can easily add LoRA and other efficient fine tuning techniques easily into huggingface models. Haven't tried it t
by rdedev 4y ago
Came across this library in the past where you can easily add LoRA and other efficient fine tuning techniques easily into huggingface models. Haven't tried it though and support for different models may be limited
https://adapterhub.ml/ https://adapterhub.ml/
- leobg 4y agoI’m wondering what the difference is between LoRA and adapters. Especially I’m wondering whether LoRA is being used for LLaMA because it is proven to be better or simply because it happened to be the first such technique that worked and now everyone just uses it. In other words: Is it worth it trying the adapter approach with models like LLaMA?
- TheFaheem 4y agoThe difference is that this inserts adapter layers on top of the model. In contrast, LoRA decomposes the model weight matrices using low-rank decomposition. So, LoRA increases finetuning performance by reducing parameter numbers whereas Adapters increases efficiency by keeping the pretrained model frozen (and only tunes a small number of parameters added to the model).