I don't think I see it mentioned in the paper but what would happen if you used the poisoned images to train the text encoder? Would that make the text encoder

I don't think I see it mentioned in the paper but what would happen if you used the poisoned images to train the text encoder? Would that make the text encoder itself resistant to at least the previously poisoned images? The images will still be titled and whatnot correctly, so unless someone is also attempting to trick scrapers, the actual captions will be correct. If this is the case can't you just compare the image classifier's and the actual caption to tell if it was poisoned.