There is much more out there than text. Audio, visual, touch, smell. Text isn't something humans directly train on, but representations of text from our senses.

There is much more out there than text. Audio, visual, touch, smell. Text isn't something humans directly train on, but representations of text from our senses. GPT-4 was trained on image data. Besides gaining understanding of image content it also showed improved language abilities over a GPT-4 trained with only text. Facebook is working on a smaller model with text, image, video, audio, lidar depth, infrared heat, and 6-axis motion data. If a GPT-4 was trained with data like that, what capabilities would it have? Rumor says we will know in a few months.