The embeddings provide the guidance for the diffusion models. So they have a direct influence. Similarity of the sentences means that they are around the same a

The embeddings provide the guidance for the diffusion models. So they have a direct influence. Similarity of the sentences means that they are around the same area in the multi dimensional space. And when they are used to guide the diffusion models, the images come out wrong. One thing is the experiments I have done are on sentence transformer models. Dalle-2 uses its own sentence embedding afaik.