3 ms·
Can you recommend any tutorials/resources about designing and training simple multi-modal models like CLIP? Or should I just be reading the papers and following
by berniep 2y ago
Can you recommend any tutorials/resources about designing and training simple multi-modal models like CLIP? Or should I just be reading the papers and following along?
- janalsncm 2y agoWell, the core idea is to train a text encoder and an image encoder jointly with in-batch negatives. In other words, a two tower model maximizing the diagonal and minimizing everything else. They have pseudo code in the paper.