3 ms·
TLDR: 1. Use alpha = 2*rank 2. Don't use too small ranks (rank=1 to 8) 3. Sensational title. Better title "LoRA works if done right" 4. Didn't test SVD init
by danielhanchen 2y ago
TLDR:
1. Use alpha = 2*rank
2. Don't use too small ranks (rank=1 to 8)
3. Sensational title. Better title "LoRA works if done right"
4. Didn't test SVD init
- Tostino 2y agoThanks for the TLDR. Yeah, pretty much fits my experience, though I mainly cared about the specific task performance I was training rather than caring about regressing unrelated tasks.
- danielhanchen 2y ago:) Oh ye from the paper it looks like if one uses alpha = 2*rank, sometimes LoRA does even better than full finetuning