7 ms·
Img2Prompt – Get prompts from stable diffusion generated images
- bheadmaster 4y agoI usually have problems coming up with prompts to generate the kind of images I want. This tool is useful for reverse-engineering prompts from the kind of images I want, then generating new ones in the same style. Very cool.
- sahil_chaudhary 4y agoGlad you like it
- martin-adams 4y agoI'm having a lot of fun dropping in my Midjourney images, getting a more detailed prompt, then putting it back into Midjourney for more interesting variations
- sahil_chaudhary 4y agoGlad you enjoy it
- OscarCunningham 4y agoSo I tried it with an image of a monkey that I often use for profile pictures (https://mathstodon.xyz/@OscarCunningham https://mathstodon.xyz/@OscarCunningham). This image wasn't made by Stable Diffusion. It gave me this prompt: > a monkey plushie on a white background, photograph taken by steve buscemi from a zoom lens, studio lighting, ultrarealistic Can someone tell me what Steve Buscemi is doing here?
- aspyct 4y agoPictures with zoom lenses, obviously :D
- badcppdev 4y agoCan you clarify whether or not Steve Buscemi actually took the photo?
- tbalsam 4y agoThis one also had me rolling, haha. Thank you, too. <3 :))))
- rf15 4y agoThis sounds a lot like the results you get from CLIP interrogation. Maybe they just use that and made another online service for it?
- sahil_chaudhary 4y agoIt is actually based on a different approach, it uses an image-captioning model finetuned on image-prompt pairs
- xkapastel 4y agoCLIP Interrogator uses BLIP, an image captioning model, as well as trying a bunch of prompts with CLIP. I guess you mean that this model uses the captioning model to generate the complete prompt? Is the code for this one available?
- sahil_chaudhary 4y agoAh yes, this model treats this purely as image captioning. The model isn't open source yet.
- sublinear 4y agoSteve Buscemi -> Parting Glances -> AIDS -> monkeys ???
- klondike_klive 4y agoThis is the thing about AI that I find simultaneously wonderful and terrifying. Something to do with me as a human, noticing a hilarious detail amidst a fathomless ocean. It affirms my humanity but the backdrop is dizzying randomness.
- the_generalist 4y agoHow do you do fellow humans?
- jabiko 4y agoMaybe it was trained on a image set of politicians. I put in a image Dr. Evil doing air quotes and it came up with "FirstName LastName from FirstName LastName in star trek the next generation ( 2005 ) ( 2 0 1 9 )". FirstName LastName being the name of a politician.
- VicVee 4y agoHis style is just so beautifully unique
- zardo 4y agoA photo taken in the style Steve Buscemi would have if he were a photographer.
- tbalsam 4y agoI'm dying laughing right now, this is phenomenal comedy gold. Thank you.
- futhey 4y agoI generated a bunch of variations and eventually got something kinda close to your source, but it seems pretty hit or miss: https://i.pica.so/b014bb2f-ca05-499a-a4da-1d178f49bb06.jpg https://i.pica.so/b014bb2f-ca05-499a-a4da-1d178f49bb06.jpg https://i.pica.so/5ee6e122-c2a4-44a7-8ae9-ecec24da0246.jpg https://i.pica.so/5ee6e122-c2a4-44a7-8ae9-ecec24da0246.jpg
- totetsu 4y agointeresting.. but my guess is it's using a big library of generated image and prompt pairs? So all its suggested prompts are right out of someones 'stable diffusion prompt cheatsheet.pdf' . That is to say overly outputting commonly known artists, and things like 'trending on deviant art'
- sahil_chaudhary 4y agoIt works by using an image-captioning model finetuned on SD prompts, so it may be outputting common known artists based on their occurrence in the training data
- smudgy 4y agoIt's not terribly bad at guessing! I made a few generic military guys for a side project and the actual prompt isn't too far from what I used.
- Der_Einzige 4y agoHow is this different from image captioning when the model used is a booru model? That's already a thing people do with making their training data for fine tuning these models.
- sahil_chaudhary 4y agoIt actually works on top of an image captioning model, SD takes in keywords as well like "artstation" and "octane render" which are not covered in standard captioning so that is why the difference between using an off-the-shelf captioning model vs this
- batterseapower 4y agoRelated - here's a fun short sci-fi story about a savant ("the prompt whisperer") who is able to intuit the prompt that was used to generate things: https://interconnected.org/home/2022/08/03/whisperer https://interconnected.org/home/2022/08/03/whisperer
- FeepingCreature 4y agoHee, cute, but predictable once the ... showed up. Honestly I think the ending was unnecessary; it destroys all subtlety. Of course, this is not actually how latent space works. It's the AI's understanding of concepts, not the inherent nature of concepts; that's why every model has its own version of "latent space". Though the understanding of latent space in the story is internally consistent; given a superintelligent image generator, you could do prompt engineering like this.
- speedgoose 4y agoIsn’t it just CLIP ? The model that made these image generation models possible. It’s good to describe a picture but it’s not reverse engineering. The predicted prompt usually has very little in common with the actual prompt. And it’s worse when you use embeddings or fine tuned models.
- yreg 4y agoI'm surprised that the results seem much better (more detailed and sometimes closer to the original prompt) than the regular CLIP interrogation (at least based on my limited experimentation). But as you say, even so, it still has little in common with the original prompt.
- Sophira 4y agoWhat's interesting to me is that it even tries to predict the prompt on images that came straight from Stable Diffusion with no editing - which is weird because such images actually do have the prompt embedded inside of them already. (At least, that's the case for me - the prompt and parameters are stored in a tEXt chunk in the PNG file, which can be read with, for example, "pngcheck -t".)
- sahil_chaudhary 4y agoTrue, images generated through some UIs have prompts in meta data, aim here is to work on images people find online with no metadata. So it doesn't try to read the metadata but actually predict a similar prompt
- CapsAdmin 4y agoThis is something specific to the automatic 1111 version. It's just a setting, but I believe it's on by default.
- jhbadger 4y agoNot just automatic 1111. Other SD forks like InvokeAI also embed the prompt in the png.
- hnarayanan 4y agoOk, this is very cool! I took one image from a series I generated, had it guess a prompt (very different from mine, but it doesn't matter) and have it regenerate an another image that captures the same feeling: https://imgur.com/a/Jz0mBej https://imgur.com/a/Jz0mBej
- sahil_chaudhary 4y agoNice, that's the aim, not to get the actual prompt back but get a prompt which can generate same style of images
- VadimPR 4y agoDoesn't seem to be working - an alternative is to use https://tinybots.net/artbot/interrogate https://tinybots.net/artbot/interrogate, which is a front-end to the crowdsourced https://stablehorde.net https://stablehorde.net network.
- MariaElisabeth8 4y ago[dead]
- sahil_chaudhary 4y agoShoutout to https://banana.dev https://banana.dev , couldn't have made this demo without their hosting
- mock-possum 4y agoyou spoke too soon, POSTs to https://www.img2prompt.io/api/banana https://www.img2prompt.io/api/banana are 504-ing
- sahil_chaudhary 4y agoThat was an issue on my end, should be fine now
- fire 4y agothis is neat, do you have any docs/posts about it? I presume it isn't on github?
- sahil_chaudhary 4y agoNot yet, looking into creating a write up on how it works and possibly open-sourcing
- vidarh 4y agoI tried it with a cat image, and it didn't quite capture the feel: https://m.galaxybound.com/@vidar/109829298036416109 https://m.galaxybound.com/@vidar/109829298036416109
- sahil_chaudhary 4y agoInteresting, it hasn’t been tested extensively for non-generated images.
- vidarh 4y agoIt called a picture of my girlfriend "unbelievably cute" so she's now a fan. And it described my profile picture on Mastodon as "my husband from the future that looks similar to travis scott and mark owen. he is also a good boy, very!!!, and beautiful!!!"... Not sure how to take that ;)
- geepytee 4y agoCan we also reverse engineer text? Would love text2prompt, super helpful to get better at prompting.
- thanatropism 4y agoHugged to death already? I often use terms like "sexy", "risque" etc. in the process of getting images that are quite sensible (like military people playing chess). I use img2img repeatedly looking for particular photo-film aesthetics and tend to accumulate prompts. Anyway, this would open me to charges of sexism (or worse "misogyny"), and makes me uneasy about using SD. Edit/OH: it generates prompts for like Excel screenshots but not for images made with the img2img model at hugginface. Fascinating.
- thanatropism 4y agoWhy does it choke on img2img creations? This is just fascinating. It gives a plausible prompt to at least a handful of real non-AI photos from DSLRs and iphones alike.
- sahil_chaudhary 4y agoThe dataset used to train this model didn’t have any img2img data so that would explain it
- thanatropism 4y agoI mean, it does mistake Frank Sinatra for Louis Armstrong -- I guess it makes errors. But it just refuses to process my img2img images. It breaks down. Why? It made me so agitated I made a gallery[0]. Granted, some of those images are strange, but others are just normal people doing normal things. [0]: https://publish.obsidian.md/zero-chroma-infinity/Image+gallery https://publish.obsidian.md/zero-chroma-infinity/Image+galle...
- dwringer 4y agoimg2img creations are probably confusing it because they are totally rearranging how the prompt emerges from what's in the latent space with respect to a new image. So if you make an image of <subject> by passing in an image of <subject>, it's going to represent <subject> in a fundamentally different way than if it relies purely on its own "imagination" for the rendition.
- mythz 4y agoYeah I wouldn't say it's very close: Tried it on last image I generated on: https://blazordiffusion.com/artifacts/50/50418_studio-ghibli.html https://blazordiffusion.com/artifacts/50/50418_studio-ghibli... Original Prompt: > Studio ghibli, rocket explosion, jungle, solar, green technology, optimist future > 8k, Bokeh effect, Cinematic Lighting, Octane Render, Iridescence, Vibrant > by Beeple, Asher Brown Durand, Dan Mumford, Greg Rutkowski, WLOP Img2Prompt: > a vehicle in the grass, colorful light dust, cinematic lighting, trending on artstation, ultra detailed, art by akihito yoshida Looks like a decent image classifier, but not useful for extracting the original stable diffusion prompt.
- sahil_chaudhary 4y agoAgreed, results not good for this style of images. I have a model training on a much bigger dataset of image-prompt pairs which should perform better on this.
- xena 4y agoThis doesn't work for anything that doesn't use the upstream default stable diffusion checkpoint. I generate a lot of images with Pastel-mix, Anything, Waifu Diffusion and Counterfeit, and none of those are giving sensible results with this tool.
- sahil_chaudhary 4y agoThe model underneath this is trained only on data from SD 1.4/5 so this would be expected. I have another model training which covers all models you mentioned which should perform well on those
- xena 4y agoI'd be interested to see the results of that. My email address is on my Hacker News profile if you want to talk there.
- thanatropism 4y agoCan I have some kind of direct contact with you? Email, Twitter DM, Telegram, heck, I would download some new kind of app just to have some conversations.
- sahil_chaudhary 4y agoYou can find my email on my HN profile
- petesergeant 4y agofolks, the AI called me "beautiful" and said I look like Chris Pratt, despite being a middle-aged and overweight computer programmer. They need to monetize this immediately.
- isoprophlex 4y agoSo, people are commenting that it's not very accurate etc., but I love it. Delightfully quirky tool for exploring prompts. Also I laughed out loud after putting a selfie into it and getting "mark zuckerberg's face reflected in a mirror, close up, realistic photo, medium shot, dslr, 4k, detailed"
- sahil_chaudhary 4y agoGlad you enjoy it! It hasn't been trained on non-generated images so is unpredictable when uploading a real photo, specially one with people in it
- msla 4y agoI tested it with an actual (albeit colorized) photograph: https://i.imgur.com/GpGG0SL.jpg https://i.imgur.com/GpGG0SL.jpg And got this prompt: > the man with the stupid face of a homeless person, portrait photography, 1 9 7 0 s, street photo, old photography, highly detailed, hyperrealistic The actual description is that she was Mary Ann Bevan (1874 - 1934) also known as Rose Wilmot, a woman who claimed the title of the ugliest woman in London as she suffered from acromegaly.
- philip-b 4y agoTbh I think that prompt is spot on.
- tbalsam 4y agoBad form, poor taste. Preferably not anywhere, but not on Hacker News, please. Thank you.
- sahil_chaudhary 4y agoThe model was trained exclusively on stable diffusion generated images, so can be unpredictable with non-generated images, specially images with people in it
- kir-gadjello 4y agoShameless plug: I have a similar open-source tool which uses locally executed pre-trained models, here https://github.com/kir-gadjello/extract_prompt https://github.com/kir-gadjello/extract_prompt
- sahil_chaudhary 4y agoNice, does this use clip-interrogator?
- kir-gadjello 4y agoYes, it's strongly influenced by clip-interrogator, but I revamped the algorithm quite a bit. I think it could be improved even further without resorting to fine-tuning the BLIP model.
- shagie 4y agoSomething similar (image to text description) from a bit ago - Seeing AI app from Microsoft (2017) - https://youtu.be/bqeQByqf_f8 https://youtu.be/bqeQByqf_f8 It's not prompt based intended to generate another one, but rather an accessibility tool. And some related videos: Seeing AI 2016 Prototype - A Microsoft research project - https://youtu.be/R2mC-NUAmMk https://youtu.be/R2mC-NUAmMk Seeing AI: Making the visual world more accessible - https://youtu.be/DybczED-GKE https://youtu.be/DybczED-GKE
- pwillia7 4y agoCan someone explain how this differs in method and efficacy to CLIP?
- deleted 4y ago[deleted]
- johtso 4y agoEnjoying playing with this! It would be great if the generated images had their prompts embedded in the metadata..
- henriquecm8 4y agoI sent a picture of Mr. Spock from the first pilot, looking back while walking on the transporter[1]. And it generated this prompt: > john cena walking on stage at television talk show, very coherent!!!!!!!!!!!!!!!!!!!!!! [1] https://i.imgur.com/L6hbWHX.jpg https://i.imgur.com/L6hbWHX.jpg
- vmarius 4y agoWhenever I try I get 500 error :/
- worldsavior 4y agoCool! I also how a project that does image captioning: https://github.com/DavidHuji/CapDec https://github.com/DavidHuji/CapDec
- isuleman 4y agoNot working.