For sure. That's why I mentioned multi-modal training, where the language model is trained on text in tandem with visual and audio training data. GPT-4 was trai

For sure. That's why I mentioned multi-modal training, where the language model is trained on text in tandem with visual and audio training data. GPT-4 was trained on both images and text and can even explain why visual jokes are funny: https://openai.com/research/gpt-4 https://openai.com/research/gpt-4 Training an LLM on video seems like the clear next step. I wonder if images and video might not encode visuospatial relations that are helpful to learning language that describes them.