There are these things called Loras (Low Ranked adaptions) which do exactly this. "A language model with billions of parameters may be LoRA fine-tuned with onl

There are these things called Loras (Low Ranked adaptions) which do exactly this. "A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters." -- Wikipedia