5 ms·

Could it be made to do that?

by Natsu 4y ago

ShamelessC 4y ago
Definitely! The model accepts CLIP embeds, so you could average the prompt embedding with the image embedding - and then do some sort of top-k selection over a few of these image embeddings to generate "variations" of an image. I think this is how DALLE-2 does it. Also, the model would benefit from being explicitly finetuned for the task of "inpainting". This is where you randomly erase parts of an image and force the model to predict what is missing based on the prompt. This allows one to edit existing images, filling in the gaps with what they need. This is technically already supported - but uses a similar "hack" as described above by sampling some noise into an existing image and starting a little bit later in the denoising schedule. Since the model wasn't finetuned for this task - a more generic method is used that doesn't work as well.
- nl 4y ago
  I've made an attempt at image embedding averaging. That is insufficient in itself but it was a completely naive implementation. It really needs to do the denoising from one image to another and vice-versa over a few iterations. Or that's what I'm working on next anyway.
  - Natsu 4y ago
    How does denoising work, exactly and why does it seem necessary to get good results?
    - ShamelessC 4y ago
      tl;dr it allows the model to focus on different frequencies of pixel-space at different time steps. Time step zero being the low frequency/low-detail features and time step 1000 being a completely “filled out”, detailed image including all the high frequency features. Doing so in discretely many steps makes the problem tractable and differentiable. Gross oversimplification, but that’s the gist.
      - Natsu 4y ago
        Different frequencies? Does that mean it's doing something like an FFT?
        ShamelessC 4y ago
        Bit of a delayed response - but no (well, not necessarily, anyways). Operating in the frequency domain is a useful technique in machine learning. With this method however, the reason that different "timesteps" amount to different frequencies is because you start with: 0: full gaussian noise 1: less gaussian noise, more generated pixels ... 998: barely any noise, mostly generated pixels 999: no noise. generation complete. It's actually the properties of gaussian noise that allow the network to learn the importance of the frequency domain indirectly. When there is a high frequency of noise, you can expect a low frequency in pixel space. When there is less noise, you can similarly expect the pixel space to now include both low _and_ high frequency details. By the end of denoising, the full range is meant to be learned (ideally, anyway). Having said that - they may also use a conversion to fourier space - I'm aware that at least nvidia does something like this in StyleGAN3.
        Natsu 4y ago
        Why does the noise being Gaussian matter? Would it even work with other types of noise like Perlin or white noise?

Could it be made to do that?

by Natsu 4y ago

Could it be made to do that?

Definitely! The model accepts CLIP embeds, so you could average the prompt embedding with the image embedding - and then do some sort of top-k selection over a few of these image embeddings to generate "variations" of an image. I think this is how DALLE-2 does it. Also, the model would benefit from being explicitly finetuned for the task of "inpainting". This is where you randomly erase parts of an image and force the model to predict what is missing based on the prompt. This allows one to edit existing images, filling in the gaps with what they need. This is technically already supported - but uses a similar "hack" as described above by sampling some noise into an existing image and starting a little bit later in the denoising schedule. Since the model wasn't finetuned for this task - a more generic method is used that doesn't work as well.

nl 4y ago
I've made an attempt at image embedding averaging. That is insufficient in itself but it was a completely naive implementation. It really needs to do the denoising from one image to another and vice-versa over a few iterations. Or that's what I'm working on next anyway.
- Natsu 4y ago
  How does denoising work, exactly and why does it seem necessary to get good results?
  - ShamelessC 4y ago
    tl;dr it allows the model to focus on different frequencies of pixel-space at different time steps. Time step zero being the low frequency/low-detail features and time step 1000 being a completely “filled out”, detailed image including all the high frequency features. Doing so in discretely many steps makes the problem tractable and differentiable. Gross oversimplification, but that’s the gist.
    - Natsu 4y ago
      Different frequencies? Does that mean it's doing something like an FFT?
      - ShamelessC 4y ago
        Bit of a delayed response - but no (well, not necessarily, anyways). Operating in the frequency domain is a useful technique in machine learning. With this method however, the reason that different "timesteps" amount to different frequencies is because you start with: 0: full gaussian noise 1: less gaussian noise, more generated pixels ... 998: barely any noise, mostly generated pixels 999: no noise. generation complete. It's actually the properties of gaussian noise that allow the network to learn the importance of the frequency domain indirectly. When there is a high frequency of noise, you can expect a low frequency in pixel space. When there is less noise, you can similarly expect the pixel space to now include both low _and_ high frequency details. By the end of denoising, the full range is meant to be learned (ideally, anyway). Having said that - they may also use a conversion to fourier space - I'm aware that at least nvidia does something like this in StyleGAN3.
        Natsu 4y ago
        Why does the noise being Gaussian matter? Would it even work with other types of noise like Perlin or white noise?