You can encode an image just like you encode a text prompt. The diffusers library does this in the image variation pipeline: https://github.com/huggingface/diff

You can encode an image just like you encode a text prompt. The diffusers library does this in the image variation pipeline: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_image_variation.py#L354 https://github.com/huggingface/diffusers/blob/main/src/diffu... And you can merge that encoded image with an encoded text prompt, if you wish. This is presumably something different if its actually altering the weights of the model. It sounds like they figured out how to run a huge learning rate without training exploding?