There's a related write-up here you might find interesting: https://wandb.ai/learning-at-home/LM_OWT/reports/Parameter-sharing-revisited-again---VmlldzoxOTAxNjc

There's a related write-up here you might find interesting: https://wandb.ai/learning-at-home/LM_OWT/reports/Parameter-sharing-revisited-again---VmlldzoxOTAxNjcx https://wandb.ai/learning-at-home/LM_OWT/reports/Parameter-s... It covers some experiments on weight tying, one of which is actually LoRA and random weights.