4 ms·

> I'm surprised they didn't cite this; it's a well known paper. I'm surprised you copied and pasted all of that without explaining what it means. Does LoRA pe

> I'm surprised they didn't cite this; it's a well known paper. I'm surprised you copied and pasted all of that without explaining what it means. Does LoRA perform worse, better or statistically insignificantly different to FullFT? You aren't able to tell from what you pasted, are you?

crimsoneer 1y ago
If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...?
- p1esk 1y ago
  The paper does not make any clear conclusions about LoRA vs FullFT performance, beyond "the two methods seem to be learning different things".
- lelanthran 1y ago
  > If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...? The answer is "There's a difference, perhaps", but the GP appeared to imply that LoRA performed worse. My understanding is that that paper found differences, but did not conclude that the differences were quantifiably better or worse, but this is not what GP's post implied.
cheald 1y ago
Standard LoRA (W_delta = B@A with standard inits) generally underperforms FT, primarily because of "intruder dimensions" (new high-ranking singular vectors which misalign with the singular vectors of the underlying weights) as outlined in the paper. There are techniques like PiCa and SVFT which can mitigate much of the loss, though.
- tangjurine 1y ago
  pica came out two days ago, how did you find out about it?
  - cheald 1y ago
    The one I was referring to was from this paper, first published in May: https://arxiv.org/abs/2505.20211v1 https://arxiv.org/abs/2505.20211v1 I don't recall how I found out about it, but it was either paperswithcode or an LLM research session working through the intruder dimensions problem. In my Stable Diffusion tests, it substantially improves LoRA training speed and fidelity, though I've got some experiments that seem to even further substantially improve on it by adding learnable rotations of the singular vectors.

crimsoneer 1y ago

If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...?

p1esk 1y ago
The paper does not make any clear conclusions about LoRA vs FullFT performance, beyond "the two methods seem to be learning different things".
lelanthran 1y ago
> If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...? The answer is "There's a difference, perhaps", but the GP appeared to imply that LoRA performed worse. My understanding is that that paper found differences, but did not conclude that the differences were quantifiably better or worse, but this is not what GP's post implied.

cheald 1y ago

Standard LoRA (W_delta = B@A with standard inits) generally underperforms FT, primarily because of "intruder dimensions" (new high-ranking singular vectors which misalign with the singular vectors of the underlying weights) as outlined in the paper. There are techniques like PiCa and SVFT which can mitigate much of the loss, though.

tangjurine 1y ago
pica came out two days ago, how did you find out about it?
- cheald 1y ago
  The one I was referring to was from this paper, first published in May: https://arxiv.org/abs/2505.20211v1 https://arxiv.org/abs/2505.20211v1 I don't recall how I found out about it, but it was either paperswithcode or an LLM research session working through the intruder dimensions problem. In my Stable Diffusion tests, it substantially improves LoRA training speed and fidelity, though I've got some experiments that seem to even further substantially improve on it by adding learnable rotations of the singular vectors.