3 ms·

What is the main difference between training and fine tuning? Can you start with a model trained only in producing the letter a, and then fine tune it to learn

by nico 3y ago

What is the main difference between training and fine tuning? Can you start with a model trained only in producing the letter a, and then fine tune it to learn b, then c, then words, sentences, etc?

worldsayshi 3y ago
Yeah, since fine tuning seems to be so much more cheaper than training why haven't OpenAI fine tuned ChatGPT on data past 2021?
- ajb117 3y ago
  My guess is that it's because they've already done RLHF on top of the standard next token prediction. In other words, they can't cheaply fine tune ChatGPT without undoing the RLHF objective by training on next token prediction with post-2021 data, and then retraining with RLHF to make sure it still gives good human-like output. I mention the "undoing RLHF" since it's not uncommon for fine-tuned models to increase in error in the original training objective after being fine-tuned with a different one. I think people saw this happen in BERT. Also ChatGPT is almost certainly huge.
- heliophobicdude 3y ago
  One argument is that it can contaminate training data from output of itself or other models. We have already documented evidence of the effect of this. In the GPT-4 technical report [1], they reported contamination of humaneval data in the training data. They did measure against a "non-contaminated" training set but no idea if that can still be trusted. Why would this matter? We can have seemingly strong benchmarks for containments but measures poorly against new and quarantined information. Classic over fitting. Another argument is that data being put out there could very much be wrong and the amounts of it amplified by other models. Take a look at this sample of demonstration data for codealpaca [2]. Not only is its output wrong but bad practices like,making up a random computation without it having access to a place to run a calculation, teaches the model these type of responses are ok. { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." } 1: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf 2: https://github.com/sahil280114/codealpaca/commit/0d265112c705f88848783dfdf9bdca9df64d8e35#diff-0585315a2292ca9c3186c41e52ce9427a39b5f595d4bee6aae15e9443c95051b https://github.com/sahil280114/codealpaca/commit/0d265112c70...
londons_explore 3y ago
Ideally you train a model right to begin with, and no fine tuning is necessary. However, sometimes you can't do that. For example, perhaps you want your model to always talk like a pirate, but you don't have billions of words spoken like a pirate to train on. So the next best thing is to train a model on all english text (which you have lots of), and then finetune on your smaller dataset of pirate speech. Finetuning is simply more training, but with a different dataset and often a different learning rate. Typically, finetuning uses far far far less data and compute, and can be done by individuals with a home PC, whereas training a large language model from scratch is in the $1M - $1B range.
swalsh 3y ago
Not an expert, but my high level understanding is this: If a model is a set of inputs, some middle layers, and a set of outputs. Fine tuning concentrates on only the output layers. Useful for taking a generic model with a base level of knowledge, and tuning it so the output is more useful for an application specific use case.
- ajb117 3y ago
  I think that's more in line with transfer learning, a variant of fine-tuning. If I'm reading this article correctly, they're fine-tuning the LMs end-to-end.
- RockyMcNuts 3y ago
  not strictly true I think - you could add new units throughout and train those while freezing existing units (adapter-based fine-tuning) - you could train all units and use e.g. low-rank adaptation to limit how much they can change - you could do prefix tuning and train an input to add at every layer see e.g. - https://lightning.ai/pages/community/article/understanding-llama-adapters/ https://lightning.ai/pages/community/article/understanding-l...
deleted 3y ago
[deleted]
Taek 3y ago
For "full fine tuning", mathematically there's no difference. Fine tuning is just extending the training on new data. What you are suggesting is called "curriculum learning", and though it hasn't been applied to LLMs yet to the best of my knowledge, it has proven to improve learning and decrease training times in other areas of ML.

by nico 3y ago

What is the main difference between training and fine tuning? Can you start with a model trained only in producing the letter a, and then fine tune it to learn b, then c, then words, sentences, etc?

worldsayshi 3y ago

Yeah, since fine tuning seems to be so much more cheaper than training why haven't OpenAI fine tuned ChatGPT on data past 2021?

ajb117 3y ago
My guess is that it's because they've already done RLHF on top of the standard next token prediction. In other words, they can't cheaply fine tune ChatGPT without undoing the RLHF objective by training on next token prediction with post-2021 data, and then retraining with RLHF to make sure it still gives good human-like output. I mention the "undoing RLHF" since it's not uncommon for fine-tuned models to increase in error in the original training objective after being fine-tuned with a different one. I think people saw this happen in BERT. Also ChatGPT is almost certainly huge.
heliophobicdude 3y ago
One argument is that it can contaminate training data from output of itself or other models. We have already documented evidence of the effect of this. In the GPT-4 technical report [1], they reported contamination of humaneval data in the training data. They did measure against a "non-contaminated" training set but no idea if that can still be trusted. Why would this matter? We can have seemingly strong benchmarks for containments but measures poorly against new and quarantined information. Classic over fitting. Another argument is that data being put out there could very much be wrong and the amounts of it amplified by other models. Take a look at this sample of demonstration data for codealpaca [2]. Not only is its output wrong but bad practices like,making up a random computation without it having access to a place to run a calculation, teaches the model these type of responses are ok. { "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." } 1: https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf 2: https://github.com/sahil280114/codealpaca/commit/0d265112c705f88848783dfdf9bdca9df64d8e35#diff-0585315a2292ca9c3186c41e52ce9427a39b5f595d4bee6aae15e9443c95051b https://github.com/sahil280114/codealpaca/commit/0d265112c70...

londons_explore 3y ago

Ideally you train a model right to begin with, and no fine tuning is necessary. However, sometimes you can't do that. For example, perhaps you want your model to always talk like a pirate, but you don't have billions of words spoken like a pirate to train on. So the next best thing is to train a model on all english text (which you have lots of), and then finetune on your smaller dataset of pirate speech. Finetuning is simply more training, but with a different dataset and often a different learning rate. Typically, finetuning uses far far far less data and compute, and can be done by individuals with a home PC, whereas training a large language model from scratch is in the $1M - $1B range.

swalsh 3y ago

Not an expert, but my high level understanding is this: If a model is a set of inputs, some middle layers, and a set of outputs. Fine tuning concentrates on only the output layers. Useful for taking a generic model with a base level of knowledge, and tuning it so the output is more useful for an application specific use case.

ajb117 3y ago
I think that's more in line with transfer learning, a variant of fine-tuning. If I'm reading this article correctly, they're fine-tuning the LMs end-to-end.
RockyMcNuts 3y ago
not strictly true I think - you could add new units throughout and train those while freezing existing units (adapter-based fine-tuning) - you could train all units and use e.g. low-rank adaptation to limit how much they can change - you could do prefix tuning and train an input to add at every layer see e.g. - https://lightning.ai/pages/community/article/understanding-llama-adapters/ https://lightning.ai/pages/community/article/understanding-l...

deleted 3y ago

[deleted]

Taek 3y ago

For "full fine tuning", mathematically there's no difference. Fine tuning is just extending the training on new data. What you are suggesting is called "curriculum learning", and though it hasn't been applied to LLMs yet to the best of my knowledge, it has proven to improve learning and decrease training times in other areas of ML.