3 ms·

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generaliz

by valine 3y ago

Training a local LLM on individual facts is a tricky one. Typically it’s not possible to train with a limited quantity of data and expect the model to generalize on that data well. In context learning generalizes well, but it’s a bad fit for an “employee” model that’s supposed to learn over a long stretch of time. If your goal is to bake new concepts into the model weights, your only real option is a dataset with that concept being used in a wide variety of contexts. A more feasible approach I think would be retrial augmented generation. You’d essentially store your conversations in a database and calculate embeddings as you go. This would allow you to later do a natural language search of the database, and insert the most relevant portion of the conversation into your context window.

hushpiper 3y ago
Yeah, I think training for facts in general is kind of problematic since you often have to overfit and the model may lose capability in other areas. I suspect that the only situations where it really makes sense to train on facts are where the facts are very nuanced and require a lot of interpretation, or more often where the facts are just so extensive that they can't be crammed effectively into the context window you're working with. Otherwise, you're better off with a vector db and a well-written prompt.
- qup 3y ago
  What if we just train it to respect facts in general, then couldn't we just supply it a list of facts? Sort of how they made chatGPT way more likely to obey requests?
  - valine 3y ago
    You can supply the model with a list of facts already, that’s not the problem. Within the context window the model is able to learn and generalize new information. Fine tuning is very unintelligent in the sense that it doesn’t take the context of the training samples into account. It’s a dumb optimizer that’s trying to minimize next token loss. Gradient descent is not beholden to the behaviors you taught in the instruct fine tune step.
abdullin 3y ago
Embeddings can be tricky. They are just an average semantic vector over a chunk of text. There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.
- valine 3y ago
  It’s not perfect, if you know of a better alternative I would genuinely love to hear about it.
  - abdullin 3y ago
    If I absolutely need to avoid hallucinations (e.g. when building marketing/sales assistants for the businesses), then I allow LLM to control and drive search for the relevant documents. On a high level: (1) give LLM ability and enough information to "expand" user query into a multiple search phrases. Search engine will use them to find most relevant fragments via a form of embedding search (2) Get highest ranking document fragments and "show" them to LLM saying: "This are the results that were found in the document database using your search phrases via embedding similarity. Refine the search" (3) Repeat that a couple of times, then rank final documents and combine them for the final answer synthesis.

by valine 3y ago

hushpiper 3y ago

Yeah, I think training for facts in general is kind of problematic since you often have to overfit and the model may lose capability in other areas. I suspect that the only situations where it really makes sense to train on facts are where the facts are very nuanced and require a lot of interpretation, or more often where the facts are just so extensive that they can't be crammed effectively into the context window you're working with. Otherwise, you're better off with a vector db and a well-written prompt.

qup 3y ago
What if we just train it to respect facts in general, then couldn't we just supply it a list of facts? Sort of how they made chatGPT way more likely to obey requests?
- valine 3y ago
  You can supply the model with a list of facts already, that’s not the problem. Within the context window the model is able to learn and generalize new information. Fine tuning is very unintelligent in the sense that it doesn’t take the context of the training samples into account. It’s a dumb optimizer that’s trying to minimize next token loss. Gradient descent is not beholden to the behaviors you taught in the instruct fine tune step.

abdullin 3y ago

Embeddings can be tricky. They are just an average semantic vector over a chunk of text. There is a high chance that a plain similarity search (dot product or cosine distance) will bring a lot of noise and junk into the request. And high noise/signal ratio in the context tends to lead to hallucinations.

valine 3y ago
It’s not perfect, if you know of a better alternative I would genuinely love to hear about it.
- abdullin 3y ago
  If I absolutely need to avoid hallucinations (e.g. when building marketing/sales assistants for the businesses), then I allow LLM to control and drive search for the relevant documents. On a high level: (1) give LLM ability and enough information to "expand" user query into a multiple search phrases. Search engine will use them to find most relevant fragments via a form of embedding search (2) Get highest ranking document fragments and "show" them to LLM saying: "This are the results that were found in the document database using your search phrases via embedding similarity. Refine the search" (3) Repeat that a couple of times, then rank final documents and combine them for the final answer synthesis.