The images are converted to an embedding space the size of token embedding space. And the model is trained on that new embedding space. A joint representation o

The images are converted to an embedding space the size of token embedding space. And the model is trained on that new embedding space. A joint representation of text and images is formed. It’s not as though the image is converted to text tokens.