4 ms·

I wonder how close the nearest match from the training data is. Was there a cheesecake that looked almost like these generated images?

layer8 4y ago
Maybe the ML model effectively implements a lossy image database with minor randomization. :)
- fxtentacle 4y ago
  Since GANs are effectively one class of denoising auto-encoders, your summary is spot-on. This type of ML model learns to effectively compress and decompress natural images by representing it as a hierarchy of convolutional features = shape templates.
  - ad404b8a372f2b9 4y ago
    I don't think it's accurate at all to characterize GANs as denoising auto-encoders, they're not even superficially similar, unless you're talking about a very specific architecture of autoencoder-based GANs like AAEs.
    - fxtentacle 4y ago
      The part of a GAN that people use to synthesize images has to be a denoising decoder. You put in 1x1px high-dimensional noise as the embedding and it'll gradually upscale and turn that noise into the 3-dimensional output image. The part of a GAN that people use for control is based on an encoder. You put in your 3-dimensional source image and it gets converted to that 1x1px high-dimensional noise in such a way that putting the noise into the encoder will produce your image again. So the network architecture/structure of a GAN is the exact same structure as you'd use for a denoising autoencoder. You might train that GAN with a different intent, but it's still a stack of upscalers and convolutions that perfectly matches the architecture inside the decoder part of a denoising auto-encoder. BTW sometimes people also explicitly call this out. The VQ part in VQGAN stands for VQVAE, the "Vector-quantized Variational AutoEncoder". And VQGAN+CLIP is what the first open source DALL-E clones were based on.
      - ad404b8a372f2b9 4y ago
        I thought that was the comparison you were drawing but it's misguided. The denoising part of a denoising autoencoder refers to the noise applied to its input, the image, not to its latent space as would be the case in the equivalency you establish between the noise distribution used as the input of a GAN's generator and the latent space of an auto-encoder (input of the decoder). Making a comparison between the generator and a decoder and their respective input spaces has some legs but not much so. An AE's latent space is constrained in an entirely different way with a regularization term that tries to minimize the divergence between the wanted distribution and the learned distribution, that's not a problem you have with a GAN since you simply draw from your wanted distribution every time. These differences lead to learned distributions in the latent space that are entirely different, with different failure modes wrt generalization and interpolation. Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all. In addition the learning signal is not the reconstruction of the input but the classification of the generated image. That's a completely different learning process, the learned distribution is different, the gradient path is different, the training is different (much more complex for a GAN), etc... I could go on. The VQGAN would fall under the clause in my initial comment about GANs which specifically use an AE in their very specific architecture, like the AAE. In the AAE's case they use a discriminator to regularize the latent space of an AE(instead of classifying the generated image), in the VQGAN they use a full autoencoder as generator, in addition to a third network for the discriminator. These do not fit the comparison you're making and do not function in general like most GANs do. So the general comparison of GANs to autoencoders remains inaccurate.
        fxtentacle 4y ago
        > The denoising part of a denoising autoencoder refers to the noise applied to its input Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser. For example https://arxiv.org/abs/2103.04192 https://arxiv.org/abs/2103.04192 > These differences lead to learned distributions in the latent space that are entirely different I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same. > Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder". That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.
- mejutoco 4y ago
  100% It would be interesting to get an input as image and get the closest prompt that generates that image. Later we could run the prompt to get the image back. That could effectively be a lossy compression of the image, exchanging space (the image) with time (cpu).
nh23423fefe 4y ago
This feels like when I made a drawing in elementary school, and someone asks if I traced it. It just feels like looking for a way to downplay what was made by making an appeal to "creativity" or "originality". But the tide never goes out on AI and computing. The capabilities will only grow more and more impressive and unassailable. When the chatbot is completely convincing is someone going to ask, "I wonder close the responses are to training text" even though no one even blinks when fathers and sons act alike. No one demands children invent new languages to prove they aren't just "randomizing samplers"
- yellowapple 4y ago
  > No one demands children invent new languages to prove they aren't just "randomizing samplers" I sure as hell do. No son of mine will be comprehensible to other humans until he's at least two years old.
- mikkergp 4y ago
  Is this just a search engine to find relevant content and remix it a bit, or can you actually create new content. These two things don't solve the same problem, and you may run into lots of copyright problems.
  - nmilo 4y ago
    You know, if you've ever done anything creative, the brain's ability to take inspiration from something and remix it (without realizing it's not original and actually just a memory) is shocking. So are our brains really just search engines to find relevant content and remix it a bit? What really is creation?
    - mikkergp 4y ago
      DJs and music producers are exceptionally creative, but if they use existing work, they have to license it.
waynesonfire 4y ago
exactly.. where is the chocolate chip cheesecake?

layer8 4y ago

Maybe the ML model effectively implements a lossy image database with minor randomization. :)

fxtentacle 4y ago
Since GANs are effectively one class of denoising auto-encoders, your summary is spot-on. This type of ML model learns to effectively compress and decompress natural images by representing it as a hierarchy of convolutional features = shape templates.
- ad404b8a372f2b9 4y ago
  I don't think it's accurate at all to characterize GANs as denoising auto-encoders, they're not even superficially similar, unless you're talking about a very specific architecture of autoencoder-based GANs like AAEs.
  - fxtentacle 4y ago
    The part of a GAN that people use to synthesize images has to be a denoising decoder. You put in 1x1px high-dimensional noise as the embedding and it'll gradually upscale and turn that noise into the 3-dimensional output image. The part of a GAN that people use for control is based on an encoder. You put in your 3-dimensional source image and it gets converted to that 1x1px high-dimensional noise in such a way that putting the noise into the encoder will produce your image again. So the network architecture/structure of a GAN is the exact same structure as you'd use for a denoising autoencoder. You might train that GAN with a different intent, but it's still a stack of upscalers and convolutions that perfectly matches the architecture inside the decoder part of a denoising auto-encoder. BTW sometimes people also explicitly call this out. The VQ part in VQGAN stands for VQVAE, the "Vector-quantized Variational AutoEncoder". And VQGAN+CLIP is what the first open source DALL-E clones were based on.
    - ad404b8a372f2b9 4y ago
      I thought that was the comparison you were drawing but it's misguided. The denoising part of a denoising autoencoder refers to the noise applied to its input, the image, not to its latent space as would be the case in the equivalency you establish between the noise distribution used as the input of a GAN's generator and the latent space of an auto-encoder (input of the decoder). Making a comparison between the generator and a decoder and their respective input spaces has some legs but not much so. An AE's latent space is constrained in an entirely different way with a regularization term that tries to minimize the divergence between the wanted distribution and the learned distribution, that's not a problem you have with a GAN since you simply draw from your wanted distribution every time. These differences lead to learned distributions in the latent space that are entirely different, with different failure modes wrt generalization and interpolation. Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all. In addition the learning signal is not the reconstruction of the input but the classification of the generated image. That's a completely different learning process, the learned distribution is different, the gradient path is different, the training is different (much more complex for a GAN), etc... I could go on. The VQGAN would fall under the clause in my initial comment about GANs which specifically use an AE in their very specific architecture, like the AAE. In the AAE's case they use a discriminator to regularize the latent space of an AE(instead of classifying the generated image), in the VQGAN they use a full autoencoder as generator, in addition to a third network for the discriminator. These do not fit the comparison you're making and do not function in general like most GANs do. So the general comparison of GANs to autoencoders remains inaccurate.
      - fxtentacle 4y ago
        > The denoising part of a denoising autoencoder refers to the noise applied to its input Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser. For example https://arxiv.org/abs/2103.04192 https://arxiv.org/abs/2103.04192 > These differences lead to learned distributions in the latent space that are entirely different I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same. > Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder". That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.
mejutoco 4y ago
100% It would be interesting to get an input as image and get the closest prompt that generates that image. Later we could run the prompt to get the image back. That could effectively be a lossy compression of the image, exchanging space (the image) with time (cpu).

nh23423fefe 4y ago

This feels like when I made a drawing in elementary school, and someone asks if I traced it. It just feels like looking for a way to downplay what was made by making an appeal to "creativity" or "originality". But the tide never goes out on AI and computing. The capabilities will only grow more and more impressive and unassailable. When the chatbot is completely convincing is someone going to ask, "I wonder close the responses are to training text" even though no one even blinks when fathers and sons act alike. No one demands children invent new languages to prove they aren't just "randomizing samplers"

yellowapple 4y ago
> No one demands children invent new languages to prove they aren't just "randomizing samplers" I sure as hell do. No son of mine will be comprehensible to other humans until he's at least two years old.
mikkergp 4y ago
Is this just a search engine to find relevant content and remix it a bit, or can you actually create new content. These two things don't solve the same problem, and you may run into lots of copyright problems.
- nmilo 4y ago
  You know, if you've ever done anything creative, the brain's ability to take inspiration from something and remix it (without realizing it's not original and actually just a memory) is shocking. So are our brains really just search engines to find relevant content and remix it a bit? What really is creation?
  - mikkergp 4y ago
    DJs and music producers are exceptionally creative, but if they use existing work, they have to license it.

waynesonfire 4y ago

exactly.. where is the chocolate chip cheesecake?