6 ms·
The vae used in stable diffusion is not ideal for compression. I think it would be better to use the vector-quantized variant (by the same authors of latent dif
by Jack000 4y ago
The vae used in stable diffusion is not ideal for compression. I think it would be better to use the vector-quantized variant (by the same authors of latent diffusion) instead of the KL variant, then store the indexes for each quantized vector using standard entropy coding algorithms.
From the paper the VQ variant also performs better overall, SD may have chosen the KL variant only to lower vram use.
- GaggiX 4y agoKL models performs better than VQ models as you can see in the latent diffusion repo by CompVis.
- Jack000 4y agojust checked the paper again and yes you're right, the KL version is better on the openimages dataset. The VQ version is better in the inpainting comparison. In this case you'd still want to use the VQ version though, it doesn't make sense to do an 8bit quantization on the KL vectors when there's an existing quantization learned through training.