3 ms·
I guess an interesting way to translate this technique to text-to-image would be to get an image from a database that matches the text query (via CLIP) and feed
by hwers 4y ago
I guess an interesting way to translate this technique to text-to-image would be to get an image from a database that matches the text query (via CLIP) and feed that + noise into a diffusion model that only does a few denoising iterations (and no clip guidance maybe). Would be a lot faster than from-scratch diffusion.
(Another way could be to redo the architecture to include a “inspired by this image” input, which is queried from an image server at inference time.) Anyone have other ideas?
- ShamelessC 4y agoThis was one of the motivations for `clip-retrieval`, a faiss index over the CLIP embeddings (CLIP ViT-L/14 to be precise) for all the captions/images in the LAION5B-Aesthetic dataset. https://rom1504.github.io/clip-retrieval https://rom1504.github.io/clip-retrieval Try the reverse image search - it can be shockingly effective. You can pretty easily rehost the index or build a lookup over your own data if you check the GitHub repo. If you don't have any data of your own, enter a query and hit that download icon to get a CSV of `URL,Caption,CLIP score`.
- DoctorOetker 4y agoI tried "a man with shopping bags stopping a tank" and was hoping to get the Tiananmen Tank Man, but I'm having no luck with variations either. EDIT: it does contain a blurry picture of the tank man and some LEGO re-enactments when I query "tiananmen tank man", but was hoping it would more intelligently deduce the picture from the description
- ShamelessC 4y agoThere is a violence filter enabled by default that might cause that issue. Could also just be a tough query - good test!
- nutanc 4y agoHave been thinking about this idea. Are there any open source implementations of diffusion models which take an image as an input?