You already do give the image model a prompt to tell it what to do. That’s not something the embedding can use independently of how the model is already using i

You already do give the image model a prompt to tell it what to do. That’s not something the embedding can use independently of how the model is already using it. In general an embedding doesn’t have intent or awareness in the way you’re looking for. “Embedding” just means one mathematical structure stuffed inside another. So for example the real number line is embedded into the Cartesian plane as each axis- that’s an embedding. Now in this case specifically, the embeddings in any kind of transformer model encode the meaning of the thing they represent into vectors (which is what the model itself actually operates on). You can train the embedding to be more useful for a particular task at inference time, which already happens.