4 ms·
That is blatantly false, these models only work as well as they do in the first place because of instruction fine tuning. The raw models are much harder to work
by phreeza 3y ago
That is blatantly false, these models only work as well as they do in the first place because of instruction fine tuning. The raw models are much harder to work with.
- superkuh 3y agoLets not conflate a model's smartness with a model's ability to be used reliably and predictably. They're very different things. Yes, fine-tuning, for instruction, or whatever you chose, will make the models easier to work with with less going off the rails. But they'll also take a big hit on perplexity and other more robust dataset completion tests. They're objectively stupider.
- never_inline 3y agoI would like to read on this. Do you have any sauce?
- smolder 3y agoCan I recommend you use the actual word "source", and let the pointless "sauce" meme die its overdue death?
- never_inline 3y agoI would like to read on this. Do you have any source?
- phreeza 3y agoInstruction fine tuning improves performance on many tasks that are not in the fine tuning dataset. See e.g. table 14 in the instructGPT paper.
- superkuh 3y agoInstruction fine tuning definitely increases the win rate in human judged comparisons but that just means it's better at generating a style humans like. Humans aren't always right. https://arxiv.org/pdf/2203.02155.pdf https://arxiv.org/pdf/2203.02155.pdf Page 56/68 in table 14 it looks like the things the fine tunes beat the base models at are basically HellaSwag and the ones that use human evaluations. Otherwise base gpt models are winning. And just before that they're discussing how the fine-tunes do cause performance regressions on page 55.
- phreeza 3y agoI completely agree that humans are not always right, but neither is the next token in a text sequence always "right",i.e. the thing that a raw LM is best at. I just object to the blanket statement that fine tuning will make a model dumber. It will mostly mean the model is not as good at the original training task, but that really doesn't mean it is "dumber" by any definition. The question then is what fine tuning does to tasks that are neither the original training task or the fine tuning task. This will depend on the fine tuning task. It seems that instruction fine tuning improves performance on tasks that involve human interaction, so I have a hard time seeing it as the model becoming dumber. Other fine tuning tasks, such as removing toxicity, may have a higher cost on unrelated tasks, so there one could say they caused the model to become dumber.
- chaxor 3y agoI believe they are referring to the original observations of code output degradation due to adding training epochs for safety. These observations were made by Microsoft working directly with OpenAI early on. They comment on it in the technical lecture for GPT-4 in the famous 'Tikz Unicorn' example, wherein the model got better and better at making the unicorn during typical training, but then regressed when training after that for better safety. It is unclear why this may be, but to speculate, it may be due to classifying large regions of the distribution as 'off limits' and therefore less structure about these regions is modeled in detail (since it is now not needed, and summed up as 'as a LLM, I cannot'). It is certainly strange that making a model less deplorable will make it worse at coding in general, but it does appear to be a real observation.
- Slava_Propanei 3y ago[dead]