Just as food for thought, here's how I would try to build this: 1. Train a OCR task that accumulates the text in its hidden state. 2. Train an image recogniti

Just as food for thought, here's how I would try to build this: 1. Train a OCR task that accumulates the text in its hidden state. 2. Train an image recognition task that accumulates the image description in its hidden state. For both of these, the state decoder is something like the usual autoregressive transformer to convert the state to a text that can be easily scored. You now have something like the CLIP embeddings that guide DallE / Imagen, etc. You temporarily freeze that while you train language tasks on the same hidden states. And then hopefully the AI model will unify the embeddings for OCR, recognition, and language input. If that happens, then the attention layers in the decoder allow the model to learn a sort of internal monolog where the image is replaced by text describing it and the OCRed text.