I think your question should be "Will pretrained generative models replace training application-specific models from scratch?" instead of your focus on supervis

I think your question should be "Will pretrained generative models replace training application-specific models from scratch?" instead of your focus on supervised learning. The model you mentioned use supervised learning, and fine tuning also requires supervised learning (i.e. the text captions associated with the images).