4 ms·
Somewhat unintuitively, it looks like training a language model on your own data usually doesn't do what people think it will do. The usual desire is to be abl
by simonw 4y ago
Somewhat unintuitively, it looks like training a language model on your own data usually doesn't do what people think it will do.
The usual desire is to be able to ask questions of your own data - and it would seem obvious that the way to do that would be to fine tune train an existing model with that extra information.
There's actually an easier (and potentially more effective?) way of achieving this: first run a search against your own data to find relevant information, then glue that together into a prompt along with the user's question and feed that to an existing language model.
I wrote about one way of building that here: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...
Open Assistant will hopefully result in a language model we can run on our own hardware (though it maybe a few years before it's feasible to do that affordable - language models are much heavier than image models like Stable Diffusion). So it can form part of this model, even without training the model on our own custom data.