3 ms·
Isn’t it just CLIP ? The model that made these image generation models possible. It’s good to describe a picture but it’s not reverse engineering. The predicte
by speedgoose 4y ago
Isn’t it just CLIP ? The model that made these image generation models possible.
It’s good to describe a picture but it’s not reverse engineering. The predicted prompt usually has very little in common with the actual prompt. And it’s worse when you use embeddings or fine tuned models.
- yreg 4y agoI'm surprised that the results seem much better (more detailed and sometimes closer to the original prompt) than the regular CLIP interrogation (at least based on my limited experimentation). But as you say, even so, it still has little in common with the original prompt.
- Sophira 4y agoWhat's interesting to me is that it even tries to predict the prompt on images that came straight from Stable Diffusion with no editing - which is weird because such images actually do have the prompt embedded inside of them already. (At least, that's the case for me - the prompt and parameters are stored in a tEXt chunk in the PNG file, which can be read with, for example, "pngcheck -t".)
- sahil_chaudhary 4y agoTrue, images generated through some UIs have prompts in meta data, aim here is to work on images people find online with no metadata. So it doesn't try to read the metadata but actually predict a similar prompt
- CapsAdmin 4y agoThis is something specific to the automatic 1111 version. It's just a setting, but I believe it's on by default.
- jhbadger 4y agoNot just automatic 1111. Other SD forks like InvokeAI also embed the prompt in the png.
- sahil_chaudhary 4y agoIt is based on an image-captioning model, so a different approach then CLIP interrogator, though you are correct that aim is not to get the exact prompt back but actually get a prompt to generate similar styles of images