4 ms·
does lora do a good job at teaching the model new things that werent in the training data? without trillions of examples of following instructions at a million
by 8note 2mo ago
does lora do a good job at teaching the model new things that werent in the training data?
without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with
- irthomasthomas 2mo agoNo. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.
- didgeoridoo 2mo agoThere was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though. [0]: https://sakana.ai/doc-to-lora/ https://sakana.ai/doc-to-lora/
- irthomasthomas 2mo agointeresting... thanks.