3 ms·
> more than a single model and a lot of finetunes/high rank LoRAs I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using ba
by londons_explore 3y ago
> more than a single model and a lot of finetunes/high rank LoRAs
I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone.
The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.
- kcorbitt 3y agoThis actually already exists! We did a writeup of the relevant optimizations here: https://openpipe.ai/blog/s-lora https://openpipe.ai/blog/s-lora
- xzyaoi 3y agoThere's also papers for hosting full-parameter fine-tuned models: https://arxiv.org/abs/2312.05215 https://arxiv.org/abs/2312.05215 Disclaimer: I'm one of the authors.
- azeirah 3y agoI recall a recent discussion about a technique to load the diff in weights between a lora and base model, zip it and transfer it on a per-needs basis.