4 ms·

> "The idea of fine-tuning has a strong research pedigree. The approach can be traced back to 2018, when two influential papers were published." The article re

by jph00 3y ago

> "The idea of fine-tuning has a strong research pedigree. The approach can be traced back to 2018, when two influential papers were published." The article refers to the BERT and GPT papers as the source of the fine-tuning idea. However, we actually first demonstrated it for universal models in 2017 and published the ULMFiT (Howard and Ruder) paper in early 2018. Prior to that, Dai and Le demonstrated the technique for in-corpus datasets. So it would be more accurate to say the approach can be traced back to those two papers, rather than to BERT and GPT. BERT and GPT showed the effectiveness of scaling up the amount of data and compute, and switching the model architecture to Transformers (amongst other things).

newhaus1994 3y ago
Yeah, my understanding is the ULMFit paper was the “genesis” Of fine-tuning in the way we mean it now.
arugulum 3y ago
As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning usually meant freezing most of a model and tuning a small layer on top of it. (ELMo also fits somewhere in here, being a kind of halfway step). ULMFiT was a relatively lesser known work but to my knowledge one of the first to do 1) LM pretraining and 2) fine-tuning all the layers, albeit with some complexity (gradual unfreezing/different learning rates for layers). GPT simplified this massively by simply tuning all weights: no nuance about it. (Also added a classifier on top). BERT took this a step further, benefiting from the larger size of BERT-large and bidirectional attention, which works really well on the NLU datasets of the time along with a classifier head. (It's not until T5 that seq2seq for general tasks became prominent again.) The idea that you tune all the weights of what was then considered a massive model was considered somewhat excessive at the time, but BERT so massively dominated every other architecture and approach at the time that everyone switched over to it. (Adapters (alongside PAL), one of the first parameter-efficient tuning methods in the Transformer era of NLP, came out shortly after.)
bravura 3y ago
Jeremy, if we want to roll up our sleeves on this, I'm pretty sure we can trace this even further. I'm pretty sure the earliest deep learning work from Bengio and Hinton (maybe even the unpublished circulating TRs from 2007) were basically: * Let's pretrain autoencoders, layer by layer, to a large depth. * Let's learn task specific fine-tuning through full backprop. * (Oh btw this is better than learning a task-specific head on the whole network (And this is briefly noted since it's obvious to us since we work with this stuff so much).) It was just that the entire deep learning approach was so radical overall that whole the fine-tuning vs. frozen features thing was buried in a ton a other new insights. And I mean radical. NeurIPS (then NIPS) wouldn't give Hinton, Bengion, and LeCun a workshop, so they organized a fucking satellite conference co-located at the same hotel as NIPS during the same time, on the topic of deep learning. And managed to dazzle everyone and steal the whole show. With that said, I agree with your assessment that these 2018 works very squarely and in a laser-focused way put the idea of fine-tuning. There is totally huge value and impact for correctly packaging and rigorously assessing a specific approach known to be good but that was sort of an after-thought in published work. by expert practitioners. With that said, I'll self-cite here and acknowledge (take the blame for ?) turning the NLP community on to using large-pretrained models and NOT fine-tuning them at all in 2010. Since the existing practice of designing features using expert knowledge hadn't gotten the community very far, I noticed Collobert + Weston doing large-scale pretraining of features in their amazing early NLP work that everyone slept on, and was like: "Hmmm, maybe NLP people would actually try deep nets if they could just use the features on their existing models and NOT worry about learning how to fine tune a net." Anyway, greetings from Berlin.
- jph00 3y ago
  Joseph your work on word vectors was really the key foundation behind modern NLP. Ulmfit and everything else builds on top of that.

by jph00 3y ago

newhaus1994 3y ago

Yeah, my understanding is the ULMFit paper was the “genesis” Of fine-tuning in the way we mean it now.

arugulum 3y ago

As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning usually meant freezing most of a model and tuning a small layer on top of it. (ELMo also fits somewhere in here, being a kind of halfway step). ULMFiT was a relatively lesser known work but to my knowledge one of the first to do 1) LM pretraining and 2) fine-tuning all the layers, albeit with some complexity (gradual unfreezing/different learning rates for layers). GPT simplified this massively by simply tuning all weights: no nuance about it. (Also added a classifier on top). BERT took this a step further, benefiting from the larger size of BERT-large and bidirectional attention, which works really well on the NLU datasets of the time along with a classifier head. (It's not until T5 that seq2seq for general tasks became prominent again.) The idea that you tune all the weights of what was then considered a massive model was considered somewhat excessive at the time, but BERT so massively dominated every other architecture and approach at the time that everyone switched over to it. (Adapters (alongside PAL), one of the first parameter-efficient tuning methods in the Transformer era of NLP, came out shortly after.)

bravura 3y ago

Jeremy, if we want to roll up our sleeves on this, I'm pretty sure we can trace this even further. I'm pretty sure the earliest deep learning work from Bengio and Hinton (maybe even the unpublished circulating TRs from 2007) were basically: * Let's pretrain autoencoders, layer by layer, to a large depth. * Let's learn task specific fine-tuning through full backprop. * (Oh btw this is better than learning a task-specific head on the whole network (And this is briefly noted since it's obvious to us since we work with this stuff so much).) It was just that the entire deep learning approach was so radical overall that whole the fine-tuning vs. frozen features thing was buried in a ton a other new insights. And I mean radical. NeurIPS (then NIPS) wouldn't give Hinton, Bengion, and LeCun a workshop, so they organized a fucking satellite conference co-located at the same hotel as NIPS during the same time, on the topic of deep learning. And managed to dazzle everyone and steal the whole show. With that said, I agree with your assessment that these 2018 works very squarely and in a laser-focused way put the idea of fine-tuning. There is totally huge value and impact for correctly packaging and rigorously assessing a specific approach known to be good but that was sort of an after-thought in published work. by expert practitioners. With that said, I'll self-cite here and acknowledge (take the blame for ?) turning the NLP community on to using large-pretrained models and NOT fine-tuning them at all in 2010. Since the existing practice of designing features using expert knowledge hadn't gotten the community very far, I noticed Collobert + Weston doing large-scale pretraining of features in their amazing early NLP work that everyone slept on, and was like: "Hmmm, maybe NLP people would actually try deep nets if they could just use the features on their existing models and NOT worry about learning how to fine tune a net." Anyway, greetings from Berlin.

jph00 3y ago
Joseph your work on word vectors was really the key foundation behind modern NLP. Ulmfit and everything else builds on top of that.