3 ms·

So the encoder takes one image as input and outputs a bunch of words describing it? Or there is some kind of word embedding that's separate from the image embed

by sp332 4y ago

So the encoder takes one image as input and outputs a bunch of words describing it? Or there is some kind of word embedding that's separate from the image embedding?

brucethemoose2 4y ago
You can encode an image just like you encode a text prompt. The diffusers library does this in the image variation pipeline: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_image_variation.py#L354 https://github.com/huggingface/diffusers/blob/main/src/diffu... And you can merge that encoded image with an encoded text prompt, if you wish. This is presumably something different if its actually altering the weights of the model. It sounds like they figured out how to run a huge learning rate without training exploding?