2 ms·
This is not really "stable-diffusion based image compression", since it only uses the VAE part of "stable diffusion", and not the denoising UNet. Technically,
by fjkdlsjflkds 4y ago
This is not really "stable-diffusion based image compression", since it only uses the VAE part of "stable diffusion", and not the denoising UNet.
Technically, this is simply "VAE-based image compression" (that uses stable diffusion v1.4's pretrained variational autoencoder) that takes the VAE representations and quantizes them.
(Note: not saying this is not interesting or useful; just that it's not what it says on the label)
Using the "denoising UNet" would make the method more computationally expensive, but probably even better (e.g., you can quantize the internal VAE representations more aggressively, since the denoising step might be able to recover the original data anyway).
- gliptic 4y agoIt is using the UNet, though.
- nl 4y agoIt does use the UNet to denoise the VAE compressed image: "The dithering of the palettized latents has introduced noise, which distorts the decoded result. But since Stable Diffusion is based on de-noising of latents, we can use the U-Net to remove the noise introduced by the dithering." The included Colab doesn't have line numbers, but you can see the code doing it: # Use Stable Diffusion U-Net to de-noise the dithered latents latents = denoise(latents) denoised_img = to_img(latents) display(denoised_img) del latents print('VAE decoding of de-noised dithered 8-bit latents') print('size: {}b = {}kB'.format(sd_bytes, sd_bytes/1024.0)) print_metrics(gt_img, denoised_img)
- fjkdlsjflkds 4y agoI stand corrected, then :) cheers.