tldr; > We ask the question, can pre-trained diffusion models’ un- derstanding of separate visual concepts be leveraged to as- sociate natural language to rele

tldr; > We ask the question, can pre-trained diffusion models’ un- derstanding of separate visual concepts be leveraged to as- sociate natural language to relevant spatial regions of an image? > An alpha-channel compositing process is used to iteratively generate segmentations for the regions of interest in a given image. The zero-shot functionality is endowed by our score distillation mechanism operating in latent space, that con- nects the language and vision modalities. I mean, this is basically pretty cool. They're masking out region in the image, then varying the background and running it through to find the best text classification for the region of the image. Overall, using off the shelf stable diffusion they can generate reasonably good automatic image segmentation (ie. this bit is a 'teddy bear', this bit is a 'hand', etc.) on images. I totally did not understand the bit about magically generating the alpha masks honestly (anyone care to explain how the MLP magically turns into a mask of a cat?) but over all, very very cool being able to use the existing model to segment images like this without having to use a massive dataset of labelled images!