I see where you're coming from, but if you take a look at the system prompts for these models (some are public, some have partially or fully leaked), you'll see

I see where you're coming from, but if you take a look at the system prompts for these models (some are public, some have partially or fully leaked), you'll see that is no longer a concern. At least not for the kind of models being discussed here. That older generation of image diffusion models (e.g. Stable Diffusion) used text encoders like CLIP [0], which simply don't have the language understanding that even the smaller modern LLMs do. Later image models moved on to using variants of T5 [1], sometimes in addition to CLIP variants (this is how FLUX.1 works). The state of the art for open models in this regard (right now, likely out of date before I can finish formatting this comment..) is probably Qwen-Image [2] which uses Qwen2.5-VL [3]. That is a multimodal LLM with native vision capabilities in addition to text. It comes in a few sizes (up to 72 billion parameters), but the one commonly used with Qwen-Image is still 7b parameters. [0]: https://openai.com/index/clip/ https://openai.com/index/clip/ [1]: https://en.wikipedia.org/wiki/T5_(language_model) https://en.wikipedia.org/wiki/T5_(language_model) [2]: https://arxiv.org/abs/2508.02324 https://arxiv.org/abs/2508.02324 [3]: https://arxiv.org/abs/2502.13923 https://arxiv.org/abs/2502.13923