6 ms·

Our humble attempt at “how much data do you need to fine-tune”

149 points by gnahzby 3y ago

Scene_Cast2 3y ago
I'm surprised that this few samples made such a big difference. Perhaps the learning rate is jacked up? Otherwise, the last couple of batches during regular training would set the overall model tone.
- 3abiton 3y ago
  One thing they did not test was the complexity of the task. I doubt with "hundred" samples it would be possible to get it to be that accurately usable. But experiments are needed to back up that claim.
TheCaptain4815 3y ago
Does the majority of the 'power' of these finetunes lay in the dataset? So once GPT4 finetuning is out (this fall), could they in theory just use the exact same datasets for GPT4 finetune?
bearjaws 3y ago
One thing I love to see here, mainly because I was going to make it myself, is the chart on latency. GPT4 is practically unusable unless you spend $10k+ a month and have an enterprise account. No real end user wants to wait 20-40 seconds for a response, only to find it was 80% of what they wanted.
- swyx 3y ago
  > ...our experiment showed that fine-tuned GPT-3.5 models were significantly faster than [base GPT-3.5] by 3.6 to 3.76 times. does this pass the sniff test? under what situations could a finetuned single tenant model be faster than a batchable multitenant base model?
  - gnahzby 3y ago
    This result really surprised us too, but it seems like we were not alone in observing this: https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gpt-3-5-vs-base-gpt-3-5-a-case-study-f3619b4f8cd8 https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gp... That being said, both accounts that we experimented on were personal accounts ( with a very limited budget) so maybe there are some special behaviors (e.g. some rate limit that only exists for base models)
  - eqian99 3y ago
    Yeah we were definitely surprised! This was a small scale experiment; we uploaded our notebook for latency tests here if anyone is interested taking a look at the code / trying it their own finetuned models: https://github.com/eqian99/finetuning-experiment https://github.com/eqian99/finetuning-experiment
  - dnnssl2 3y ago
    Under the same conditions where enterprise versions of the API have significantly less latency and better reliability than personal. OpenAI can change anything about the underlying infrastructure.
joewferrara 3y ago
They test two fine tuning tasks in the article - reliable output formatting and custom tone. These are two tasks (reliable output formatting in particular) that are advertised regularly as areas where fine tuning an LLM should work. The goal is not to change what the LLM knows, but to change how the LLM communicates what it knows. In theory the user wants to leverage the LLMs knowledge base and using the different output format is more useful to the user. The hard question IMO is the question of when does it make sense to fine tune an LLM to update it's knowledge and how much data is needed in this case? I have not seen anyone show a real example of succeeded in this case and wonder if it's close to as difficult as training the LLM from scratch or if it's a feasible fine tuning use case.
- KRAKRISMOTT 3y ago
  Why not use the finite state machine logits filter trick for controlling output format? It seems to work pretty well for JSON
  - antman 3y ago
    What is that? Sounds useful
    - evrimoztamur 3y ago
      I think he's referring to https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Controlling token outputs to match a regex generates valid JSON 100% of the time!
- jawerty 3y ago
  A lot of it is based on what you’re aiming to generalize. Typically for data that is fairly “conceptual” I find a DPR type solution to be better for most people with a similarity search. But if you want to generalize on style/data extraction/basic pattern matching fine tuning is incredible. I talk through some of these concepts on a fine tuning llama 2 live stream I did a couple months ago https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD
tomohelix 3y ago
Is this something like short term vs long term memory? The context window for LLMs is its short term memory where you can tell it to do things or quickly define something and the LLM can learn very quickly even with just 1 example or a sentence. But it forgets immediately once the work is done. But for finetuning, it commits the knowledge into its weight network and have a "deeper" understanding? The cost is it takes more effort and energy to do so? If so, let say in the future, we have an LLM with 100K token context windows but with a subsystem where it notices some knowledge keeps being repeated in the context and then store that knowledge for finetuning when the LLM is not doing inference. Basically a mirror of the way we human work? Is that possible? An LLM that constantly improved and can adapt to new knowledge?
- BoorishBears 3y ago
  Fine tuning is mostly useless for direct addition of knowledge. You can use it to improve knowledge in indirect ways: - get the model better at crafting queries for an external data source - get the model better a tool usage to do computation with an external system - get more useful embeddings from BERT/SBERT - tell the model what it cannot answer accurately But in general, fine tuning is noise right now because 99% of the people chasing it actually don't need it. If you want to change how the model presents text, use fine tuning. If you want to change what the model can present, fine tuning is a hopeless way of doing it.
  - CMCDragonkai 3y ago
    > If you want to change what the model can present, fine tuning is a hopeless way of doing it. If this is what I want, then what is the right way to do it?
    - BoorishBears 3y ago
      Put the information in the context window of the model. If you're answering questions on current events, hook it up to a news API and let it take in a user query, return a search to you, and give the result back to the LLM before it responds. Right now LLMs perform best for factual information if you treat them like flexible glue between the user and your data sources. Any knowledge you leverage from the LLM itself should "static" knowledge that can hold true across all user queries. For example, you can use the LLM's knowledge of marketing to produce marketing copy via chain of thought. Trends in marketing copy can change, but the actual act of making marketing copy takes a static set of knowledge. _ Don't use the LLM to surface competitors for the product, because one person will ask about 'Coke' and it'll say 'Pepsi', and another will name their founded in 2023 company and it'll hallucinate: especially if you're asking for structured output Instead let the LLM form some searches, take the results and let the LLM use chain of thought on what defines a competitor, and surface that result

149 points by gnahzby 3y ago

I'm surprised that this few samples made such a big difference. Perhaps the learning rate is jacked up? Otherwise, the last couple of batches during regular training would set the overall model tone.

3abiton 3y ago
One thing they did not test was the complexity of the task. I doubt with "hundred" samples it would be possible to get it to be that accurately usable. But experiments are needed to back up that claim.

TheCaptain4815 3y ago

Does the majority of the 'power' of these finetunes lay in the dataset? So once GPT4 finetuning is out (this fall), could they in theory just use the exact same datasets for GPT4 finetune?

bearjaws 3y ago

One thing I love to see here, mainly because I was going to make it myself, is the chart on latency. GPT4 is practically unusable unless you spend $10k+ a month and have an enterprise account. No real end user wants to wait 20-40 seconds for a response, only to find it was 80% of what they wanted.

swyx 3y ago
> ...our experiment showed that fine-tuned GPT-3.5 models were significantly faster than [base GPT-3.5] by 3.6 to 3.76 times. does this pass the sniff test? under what situations could a finetuned single tenant model be faster than a batchable multitenant base model?
- gnahzby 3y ago
  This result really surprised us too, but it seems like we were not alone in observing this: https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gpt-3-5-vs-base-gpt-3-5-a-case-study-f3619b4f8cd8 https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gp... That being said, both accounts that we experimented on were personal accounts ( with a very limited budget) so maybe there are some special behaviors (e.g. some rate limit that only exists for base models)
- eqian99 3y ago
  Yeah we were definitely surprised! This was a small scale experiment; we uploaded our notebook for latency tests here if anyone is interested taking a look at the code / trying it their own finetuned models: https://github.com/eqian99/finetuning-experiment https://github.com/eqian99/finetuning-experiment
- dnnssl2 3y ago
  Under the same conditions where enterprise versions of the API have significantly less latency and better reliability than personal. OpenAI can change anything about the underlying infrastructure.

joewferrara 3y ago

They test two fine tuning tasks in the article - reliable output formatting and custom tone. These are two tasks (reliable output formatting in particular) that are advertised regularly as areas where fine tuning an LLM should work. The goal is not to change what the LLM knows, but to change how the LLM communicates what it knows. In theory the user wants to leverage the LLMs knowledge base and using the different output format is more useful to the user. The hard question IMO is the question of when does it make sense to fine tune an LLM to update it's knowledge and how much data is needed in this case? I have not seen anyone show a real example of succeeded in this case and wonder if it's close to as difficult as training the LLM from scratch or if it's a feasible fine tuning use case.

KRAKRISMOTT 3y ago
Why not use the finite state machine logits filter trick for controlling output format? It seems to work pretty well for JSON
- antman 3y ago
  What is that? Sounds useful
  - evrimoztamur 3y ago
    I think he's referring to https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Controlling token outputs to match a regex generates valid JSON 100% of the time!
jawerty 3y ago
A lot of it is based on what you’re aiming to generalize. Typically for data that is fairly “conceptual” I find a DPR type solution to be better for most people with a similarity search. But if you want to generalize on style/data extraction/basic pattern matching fine tuning is incredible. I talk through some of these concepts on a fine tuning llama 2 live stream I did a couple months ago https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD

tomohelix 3y ago

Is this something like short term vs long term memory? The context window for LLMs is its short term memory where you can tell it to do things or quickly define something and the LLM can learn very quickly even with just 1 example or a sentence. But it forgets immediately once the work is done. But for finetuning, it commits the knowledge into its weight network and have a "deeper" understanding? The cost is it takes more effort and energy to do so? If so, let say in the future, we have an LLM with 100K token context windows but with a subsystem where it notices some knowledge keeps being repeated in the context and then store that knowledge for finetuning when the LLM is not doing inference. Basically a mirror of the way we human work? Is that possible? An LLM that constantly improved and can adapt to new knowledge?

BoorishBears 3y ago
Fine tuning is mostly useless for direct addition of knowledge. You can use it to improve knowledge in indirect ways: - get the model better at crafting queries for an external data source - get the model better a tool usage to do computation with an external system - get more useful embeddings from BERT/SBERT - tell the model what it cannot answer accurately But in general, fine tuning is noise right now because 99% of the people chasing it actually don't need it. If you want to change how the model presents text, use fine tuning. If you want to change what the model can present, fine tuning is a hopeless way of doing it.
- CMCDragonkai 3y ago
  > If you want to change what the model can present, fine tuning is a hopeless way of doing it. If this is what I want, then what is the right way to do it?
  - BoorishBears 3y ago
    Put the information in the context window of the model. If you're answering questions on current events, hook it up to a news API and let it take in a user query, return a search to you, and give the result back to the LLM before it responds. Right now LLMs perform best for factual information if you treat them like flexible glue between the user and your data sources. Any knowledge you leverage from the LLM itself should "static" knowledge that can hold true across all user queries. For example, you can use the LLM's knowledge of marketing to produce marketing copy via chain of thought. Trends in marketing copy can change, but the actual act of making marketing copy takes a static set of knowledge. _ Don't use the LLM to surface competitors for the product, because one person will ask about 'Coke' and it'll say 'Pepsi', and another will name their founded in 2023 company and it'll hallucinate: especially if you're asking for structured output Instead let the LLM form some searches, take the results and let the LLM use chain of thought on what defines a competitor, and surface that result