3 ms·

I thought that was the comparison you were drawing but it's misguided. The denoising part of a denoising autoencoder refers to the noise applied to its input,

I thought that was the comparison you were drawing but it's misguided. The denoising part of a denoising autoencoder refers to the noise applied to its input, the image, not to its latent space as would be the case in the equivalency you establish between the noise distribution used as the input of a GAN's generator and the latent space of an auto-encoder (input of the decoder). Making a comparison between the generator and a decoder and their respective input spaces has some legs but not much so. An AE's latent space is constrained in an entirely different way with a regularization term that tries to minimize the divergence between the wanted distribution and the learned distribution, that's not a problem you have with a GAN since you simply draw from your wanted distribution every time. These differences lead to learned distributions in the latent space that are entirely different, with different failure modes wrt generalization and interpolation. Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all. In addition the learning signal is not the reconstruction of the input but the classification of the generated image. That's a completely different learning process, the learned distribution is different, the gradient path is different, the training is different (much more complex for a GAN), etc... I could go on. The VQGAN would fall under the clause in my initial comment about GANs which specifically use an AE in their very specific architecture, like the AAE. In the AAE's case they use a discriminator to regularize the latent space of an AE(instead of classifying the generated image), in the VQGAN they use a full autoencoder as generator, in addition to a third network for the discriminator. These do not fit the comparison you're making and do not function in general like most GANs do. So the general comparison of GANs to autoencoders remains inaccurate.

fxtentacle 4y ago
> The denoising part of a denoising autoencoder refers to the noise applied to its input Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser. For example https://arxiv.org/abs/2103.04192 https://arxiv.org/abs/2103.04192 > These differences lead to learned distributions in the latent space that are entirely different I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same. > Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder". That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.

fxtentacle 4y ago

> The denoising part of a denoising autoencoder refers to the noise applied to its input Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser. For example https://arxiv.org/abs/2103.04192 https://arxiv.org/abs/2103.04192 > These differences lead to learned distributions in the latent space that are entirely different I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same. > Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder". That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.