4 ms·
I think the QLoRA paper https://arxiv.org/pdf/2305.14314 https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers
by danielhanchen 2y ago
I think the QLoRA paper https://arxiv.org/pdf/2305.14314 https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers.
Other papers show finetuning a select few layers can also work well.
- 3abiton 2y agoAny real world performance comparison between QLoRa and LoRa?
- danielhanchen 2y agoThe QLoRA paper itself provided some cool benchmarks across many many experiments - QLoRA is near equivalent to LoRA, with it sometimes exceeding or losing 1-2% accuracy (it depends on the use case)