Well, the core idea is to train a text encoder and an image encoder jointly with in-batch negatives. In other words, a two tower model maximizing the diagonal a

Well, the core idea is to train a text encoder and an image encoder jointly with in-batch negatives. In other words, a two tower model maximizing the diagonal and minimizing everything else. They have pseudo code in the paper.