Usually you stick a CNN (or whatever other vision encoding neural network) after the image inputs and pipe the output of the CNN as an additional token or token

Usually you stick a CNN (or whatever other vision encoding neural network) after the image inputs and pipe the output of the CNN as an additional token or tokens of the GPT transformer (into the layer after the text embedding layer). That way everything is differentiable. You can pre-train the CNN and Transformer independently and then train them together (usually easier to train this way), or just train them from scratch. There are lots of other ways to combine two networks also. So input is now one (or more) images and text, and output is text. There are ways to position the input image in a particular location within the input text as well. Training data can come from websites, etc.