>But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc.

>But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc. This yields tokens None of that is necessary for an Autoregressive Transformer. You can train the transformer to predict text tokens given interleaved image and text input tokens in the context window. Google have already told us how this works. Read the Flamingo or Pali papers. You are wrong. Very wrong. It's incredible that people will crucify LLMs for "hallucinating" but then there are humans like you running around.