Jeremy, if we want to roll up our sleeves on this, I'm pretty sure we can trace this even further. I'm pretty sure the earliest deep learning work from Bengio

Jeremy, if we want to roll up our sleeves on this, I'm pretty sure we can trace this even further. I'm pretty sure the earliest deep learning work from Bengio and Hinton (maybe even the unpublished circulating TRs from 2007) were basically: * Let's pretrain autoencoders, layer by layer, to a large depth. * Let's learn task specific fine-tuning through full backprop. * (Oh btw this is better than learning a task-specific head on the whole network (And this is briefly noted since it's obvious to us since we work with this stuff so much).) It was just that the entire deep learning approach was so radical overall that whole the fine-tuning vs. frozen features thing was buried in a ton a other new insights. And I mean radical. NeurIPS (then NIPS) wouldn't give Hinton, Bengion, and LeCun a workshop, so they organized a fucking satellite conference co-located at the same hotel as NIPS during the same time, on the topic of deep learning. And managed to dazzle everyone and steal the whole show. With that said, I agree with your assessment that these 2018 works very squarely and in a laser-focused way put the idea of fine-tuning. There is totally huge value and impact for correctly packaging and rigorously assessing a specific approach known to be good but that was sort of an after-thought in published work. by expert practitioners. With that said, I'll self-cite here and acknowledge (take the blame for ?) turning the NLP community on to using large-pretrained models and NOT fine-tuning them at all in 2010. Since the existing practice of designing features using expert knowledge hadn't gotten the community very far, I noticed Collobert + Weston doing large-scale pretraining of features in their amazing early NLP work that everyone slept on, and was like: "Hmmm, maybe NLP people would actually try deep nets if they could just use the features on their existing models and NOT worry about learning how to fine tune a net." Anyway, greetings from Berlin.