I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something

I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something similar), the next step would be to actually predict image tokens using the LM loss, this way it would be able to use the knowledge gained by "seeing" on other tasks like: making actual good ASCII art, making SVG that makes sense, and on a less superficial level having a better world model.