4 ms·

Could someone explain in simple terms exactly what fine-tuning does? Does it show the model how to answer questions, or does it give it new information, or bot

by bambax 3y ago

Could someone explain in simple terms exactly what fine-tuning does? Does it show the model how to answer questions, or does it give it new information, or both? Is there a way to restrict answers to the fine-tuned data? For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? If yes, how does one reduce hallucination? And would that perform better than feeding possible source documents as part of the prompt every time?

skerit 3y ago
> For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? I initially also thought this would be one of the best use cases for fine-tuning (teaching the model new data), but I've seen quite a few people say fine-tuning should not be used to teach the model new data, but more like new formatting and style of response. This blog post seems to concur. I do wonder how OpenAI does fine-tuning. I'm guessing it doesn't use Lora.
- textcortex 3y ago
  Most probably they are doing something called prompt tuning. This creates a small ai model that adds virtual tokens to prompt before passing to original model: https://developer.nvidia.com/blog/an-introduction-to-large-language-models-prompt-engineering-and-p-tuning/ https://developer.nvidia.com/blog/an-introduction-to-large-l...
tedsanders 3y ago
>Could someone explain in simple terms exactly what fine-tuning does? Fine-tuning shows the model examples of sequences it should produce. The model is updated to become more likely to produce sequences like those examples. What precisely 'like those examples' means for brand new prompts unlike those in the training distribution is the black magic of generalization. >Does it show the model how to answer questions, or does it give it new information, or both? It can be used to teach style, or information, or both. >Is there a way to restrict answers to the fine-tuned data? There is no foolproof way to restrict answers to fine-tuned data. You might be able to approach decent performance if you show it examples of refusing on all topics not related to X. >For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? Short answer: I wouldn't recommend fine-tuning. Long answer: it depends on your task, your expertise, and your tolerance for collecting large datasets and iterating. I generally recommend retrieval. Putting info in the input has a few advantages over fine-tuning: you can check where information is coming from, and it's easier for the model to answer without hallucinating (akin to a student taking a test with open notes they can refer to, rather than trying to remember a textbook they read a week ago). Retrieval is best at lookup type questions and is worse at questions requiring comparisons or mixing of many pieces of source data; possible fine-tuning has some edge there.
- bambax 3y ago
  Thank you for this answer! > I generally recommend retrieval Yes that's what everyone's saying and it's also what we're working on. I was wondering what fine-tuning may be used for. Are there use cases where fine-tuning might be worth it (esp; given all the hard work it entails)? > akin to a student taking a test with open notes they can refer to, rather than trying to remember a textbook they read a week ago Excellent analogy! Thanks!
  - irthomasthomas 3y ago
    Yes, this is for training your own public facing chatbot. So that it does not easily break character and go wildly off-script, like bing, and start ranting at the old lady trying to check her balance. This will likely dramatically increase the number of chatgpt customer service bots in the near future, but on the up side, at least they wont all be regurgitating that "as a language model" shtick. That is if they are willing to pay to beat it out him.
rchtwlm 3y ago
>For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? No, it does not. Language models are not for storing or accessing data, as you have noticed when you refer to hallucination. If you wish to store and access data, use embeddings + a vector database. Fine tuning is for changing what kind of language the model generates. For example, if you want an AI that writes like a journalist you fine tune it on newspaper articles. If you want an AI that writes reviews, you fine tune it on reviews. And so on.
- bambax 3y ago
  Ok, thanks, but do you really need fine tuning for that, can't you tell the model in the prompt?
  - hnuser123456 3y ago
    It seems like LLMs can easily start to "forget" requests to modify their style or tone if it's a small part of the overall context, and they drift back towards the style or tone they naturally have (or were encouraged to have by their original trainers.)
kuida0r3 3y ago
Andrew Ng's deeplearning.ai just released a course on this yesterday https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... - at a high level it lets one put more data into the model than fits into the prompt. And yes your use case of a large private knowledge base is one of the prime examples she used in the course. Scenarios that are domain-specific or privacy conscious probably makes more sense for finetuning as opposed to prompting.
- OthmaneHamzaoui 3y ago
  But can't you still achieve domain-specific or privacy conscious cases by using retrieval + vector dbs?
  - kuida0r3 3y ago
    Not an expert but my understanding is retrieval+vector db enables accurate information retrieval, whereas finetuning is more focused on specific task performance and generating coherent text (maybe in a different style) generally.
ofermend 3y ago
It's important to note that this fine-tuning is what is known as "supervised fine-tuning" where you give the LLM a set of question/answer pairs and it tunes to those (see https://huyenchip.com/2023/05/02/rlhf.html https://huyenchip.com/2023/05/02/rlhf.html). This is quite different then fine-tuning the base model or doing RLHF (reinforcement learning from human feedback). My guess is that it could work well to adjust the goal of the LLM, i.e. tell it to behave in a certain way, or do a different task than just being a generalist chat bot. This is quite different than adding knowledge to the bot (known as grounded-generation or retrieval-augmented-generation), which aims to augment the base model with new data (e.g. your confidential data). So in short - I think it's not appropriate for answering questions about a large private knowledge base and GG/RAG is better suited. (if you're interested, I wrote a blog article about this recently: https://vectara.com/fine-tuning-vs-grounded-generation/ https://vectara.com/fine-tuning-vs-grounded-generation/)

by bambax 3y ago

skerit 3y ago

> For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? I initially also thought this would be one of the best use cases for fine-tuning (teaching the model new data), but I've seen quite a few people say fine-tuning should not be used to teach the model new data, but more like new formatting and style of response. This blog post seems to concur. I do wonder how OpenAI does fine-tuning. I'm guessing it doesn't use Lora.

textcortex 3y ago
Most probably they are doing something called prompt tuning. This creates a small ai model that adds virtual tokens to prompt before passing to original model: https://developer.nvidia.com/blog/an-introduction-to-large-language-models-prompt-engineering-and-p-tuning/ https://developer.nvidia.com/blog/an-introduction-to-large-l...

tedsanders 3y ago

>Could someone explain in simple terms exactly what fine-tuning does? Fine-tuning shows the model examples of sequences it should produce. The model is updated to become more likely to produce sequences like those examples. What precisely 'like those examples' means for brand new prompts unlike those in the training distribution is the black magic of generalization. >Does it show the model how to answer questions, or does it give it new information, or both? It can be used to teach style, or information, or both. >Is there a way to restrict answers to the fine-tuned data? There is no foolproof way to restrict answers to fine-tuned data. You might be able to approach decent performance if you show it examples of refusing on all topics not related to X. >For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? Short answer: I wouldn't recommend fine-tuning. Long answer: it depends on your task, your expertise, and your tolerance for collecting large datasets and iterating. I generally recommend retrieval. Putting info in the input has a few advantages over fine-tuning: you can check where information is coming from, and it's easier for the model to answer without hallucinating (akin to a student taking a test with open notes they can refer to, rather than trying to remember a textbook they read a week ago). Retrieval is best at lookup type questions and is worse at questions requiring comparisons or mixing of many pieces of source data; possible fine-tuning has some edge there.

bambax 3y ago
Thank you for this answer! > I generally recommend retrieval Yes that's what everyone's saying and it's also what we're working on. I was wondering what fine-tuning may be used for. Are there use cases where fine-tuning might be worth it (esp; given all the hard work it entails)? > akin to a student taking a test with open notes they can refer to, rather than trying to remember a textbook they read a week ago Excellent analogy! Thanks!
- irthomasthomas 3y ago
  Yes, this is for training your own public facing chatbot. So that it does not easily break character and go wildly off-script, like bing, and start ranting at the old lady trying to check her balance. This will likely dramatically increase the number of chatgpt customer service bots in the near future, but on the up side, at least they wont all be regurgitating that "as a language model" shtick. That is if they are willing to pay to beat it out him.

rchtwlm 3y ago

>For example, if one would want to use an LLM to answer questions regarding a large, private knowledge base, would it make sense to fine-tune a model on this knowledge base? No, it does not. Language models are not for storing or accessing data, as you have noticed when you refer to hallucination. If you wish to store and access data, use embeddings + a vector database. Fine tuning is for changing what kind of language the model generates. For example, if you want an AI that writes like a journalist you fine tune it on newspaper articles. If you want an AI that writes reviews, you fine tune it on reviews. And so on.

bambax 3y ago
Ok, thanks, but do you really need fine tuning for that, can't you tell the model in the prompt?
- hnuser123456 3y ago
  It seems like LLMs can easily start to "forget" requests to modify their style or tone if it's a small part of the overall context, and they drift back towards the style or tone they naturally have (or were encouraged to have by their original trainers.)

kuida0r3 3y ago

Andrew Ng's deeplearning.ai just released a course on this yesterday https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... - at a high level it lets one put more data into the model than fits into the prompt. And yes your use case of a large private knowledge base is one of the prime examples she used in the course. Scenarios that are domain-specific or privacy conscious probably makes more sense for finetuning as opposed to prompting.

OthmaneHamzaoui 3y ago
But can't you still achieve domain-specific or privacy conscious cases by using retrieval + vector dbs?
- kuida0r3 3y ago
  Not an expert but my understanding is retrieval+vector db enables accurate information retrieval, whereas finetuning is more focused on specific task performance and generating coherent text (maybe in a different style) generally.

ofermend 3y ago

It's important to note that this fine-tuning is what is known as "supervised fine-tuning" where you give the LLM a set of question/answer pairs and it tunes to those (see https://huyenchip.com/2023/05/02/rlhf.html https://huyenchip.com/2023/05/02/rlhf.html). This is quite different then fine-tuning the base model or doing RLHF (reinforcement learning from human feedback). My guess is that it could work well to adjust the goal of the LLM, i.e. tell it to behave in a certain way, or do a different task than just being a generalist chat bot. This is quite different than adding knowledge to the bot (known as grounded-generation or retrieval-augmented-generation), which aims to augment the base model with new data (e.g. your confidential data). So in short - I think it's not appropriate for answering questions about a large private knowledge base and GG/RAG is better suited. (if you're interested, I wrote a blog article about this recently: https://vectara.com/fine-tuning-vs-grounded-generation/ https://vectara.com/fine-tuning-vs-grounded-generation/)