Nowadays VLMs are kind of Frankenstein models where the vision model is separated from the LLM and the communication is only one way where the vision embeddings

Nowadays VLMs are kind of Frankenstein models where the vision model is separated from the LLM and the communication is only one way where the vision embeddings are projected into textual embeddings and aligned; I think that this method severely limits the visual capability of the model and I think that it would be fixed by training a single decoder-only transformer on both textual and image tokens, I think that this method would allow some new incredible emergent behaviors like accurately translating images of logos to SVG, actually good ascii art, more meaningful openscad scripts and the ability to generate images with more meaningful composition and text (for now models can only generate meaningful text if it's available in the prompt, this is true even with Dalle 3 as the prompt is rewritten by ChatGPT).