3 ms·

>Our method is similar to GANs in that a critic is jointly trained with the generator to minimize a divergence between the real and fake distributions, but diff

>Our method is similar to GANs in that a critic is jointly trained with the generator to minimize a divergence between the real and fake distributions, but differs in that our training does not play an adversarial game that may cause training instability, and our critic can fully leverage the weights of a pretrained diffusion model. Very glad to see GANs (or GAN-likes) coming back! Also I don't know if the examples were cherrypicked but a lot of the images look more realistic than SD. For example the dog in the snow[0] is severely oversaturated in the SD version giving it that distinct AI look (sometimes referred to as AIslop). Also the landscape scene[1] has a random lens flare in the SD version which makes it look oversaturated as well. The MIT images are much better in this regard. [0] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0009.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0008.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... [1] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0001.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0005.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_...

bosco_mcnasty 3y ago
>Very glad to see GANs (or GAN-likes) coming back! Adobe has had GigaGAN out a while now though?
amelius 3y ago
Can GANs prevent the ">10 fingers" problem?
- fzliu 3y ago
  The >10 fingers problem should be solved via higher quality data or conditioning rather than a better model architecture.
  - onion2k 3y ago
    Are you suggesting AI generates images of people with more than 10 fingers because there are too many pictures of people with more than 10 fingers in the input data? That seems unlikely..
    - jobigoud 3y ago
      Higher quality could also mean the description attached to the original images in the dataset. If you were to describe a picture of someone you would never specifically call out the fact that they have 5 fingers per hand, we take it for granted, so that kind of information may never appear in the dataset. But I think what the grant parent means by conditioning is non-textual conditioning, like ControlNet. This will always be more powerful than trying to describe something by text. Think about the description of a character in a novel vs the movie adaptation.
    - Drakim 3y ago
      In some pictures not all 5 fingers on a hand are visible, so maybe it appears that humans have a variable amount of fingers?
      - josefx 3y ago
        Humans with less than 5 fingers per hand do indeed exist. How does that lead to a default of 7?
        itishappy 3y ago
        Counting fingers is how humans do it, not necessarily how AI does. On a five-fingered hand, fingers are more likely to have neighbors than not. Why wouldn't the default be infinite fingers?
        Drakim 3y ago
        I meant more because people are holding objects or their hand is just at an angle where all fingers are not visible in the picture. I don't understand how neutral networks operate, but my layman's guess is that when you sometimes see hands with 5 fingers visible, sometimes 4, sometimes 3, sometimes 2, sometimes 1, and sometimes 0, then it's not immediately apparent that it means that every hand has between 5 and 0 fingers. Think of it this way, if the AI has ever only seen houses with a maximum of 10 windows in it's entire training set, is it so unthinkable that it sometimes draws a house with 12 windows? that's a "sensible" understanding about how houses have a variable amount of windows. It just doesn't work for fingers. I'm sure the same issue would arise if humans had other body parts that came in large quantities, but almost everything else is either 1 or 2 like the nose or eyes.
- samus 3y ago
  A lot of the problems of Stable Diffusion arise because of using a text encoder that isn't up to the task to encode the meaning sufficiently precise. For example, if you mention the word "green" in the prompt, it often carries over to the whole image instead of the object that is being described. Numbers suffer from a similar problem. Rendering text is compromised by tokenization. One problem with hands might be that they are comparatively small. The model easily gets the big picture right (head, arms, legs), but hands are so-called high-frequency details and are additionally featured in lots of different positions, which are seldom sufficiently described in the captions of the training data.

bosco_mcnasty 3y ago

>Very glad to see GANs (or GAN-likes) coming back! Adobe has had GigaGAN out a while now though?

amelius 3y ago

Can GANs prevent the ">10 fingers" problem?

fzliu 3y ago
The >10 fingers problem should be solved via higher quality data or conditioning rather than a better model architecture.
- onion2k 3y ago
  Are you suggesting AI generates images of people with more than 10 fingers because there are too many pictures of people with more than 10 fingers in the input data? That seems unlikely..
  - jobigoud 3y ago
    Higher quality could also mean the description attached to the original images in the dataset. If you were to describe a picture of someone you would never specifically call out the fact that they have 5 fingers per hand, we take it for granted, so that kind of information may never appear in the dataset. But I think what the grant parent means by conditioning is non-textual conditioning, like ControlNet. This will always be more powerful than trying to describe something by text. Think about the description of a character in a novel vs the movie adaptation.
  - Drakim 3y ago
    In some pictures not all 5 fingers on a hand are visible, so maybe it appears that humans have a variable amount of fingers?
    - josefx 3y ago
      Humans with less than 5 fingers per hand do indeed exist. How does that lead to a default of 7?
      - itishappy 3y ago
        Counting fingers is how humans do it, not necessarily how AI does. On a five-fingered hand, fingers are more likely to have neighbors than not. Why wouldn't the default be infinite fingers?
      - Drakim 3y ago
        I meant more because people are holding objects or their hand is just at an angle where all fingers are not visible in the picture. I don't understand how neutral networks operate, but my layman's guess is that when you sometimes see hands with 5 fingers visible, sometimes 4, sometimes 3, sometimes 2, sometimes 1, and sometimes 0, then it's not immediately apparent that it means that every hand has between 5 and 0 fingers. Think of it this way, if the AI has ever only seen houses with a maximum of 10 windows in it's entire training set, is it so unthinkable that it sometimes draws a house with 12 windows? that's a "sensible" understanding about how houses have a variable amount of windows. It just doesn't work for fingers. I'm sure the same issue would arise if humans had other body parts that came in large quantities, but almost everything else is either 1 or 2 like the nose or eyes.
samus 3y ago
A lot of the problems of Stable Diffusion arise because of using a text encoder that isn't up to the task to encode the meaning sufficiently precise. For example, if you mention the word "green" in the prompt, it often carries over to the whole image instead of the object that is being described. Numbers suffer from a similar problem. Rendering text is compromised by tokenization. One problem with hands might be that they are comparatively small. The model easily gets the big picture right (head, arms, legs), but hands are so-called high-frequency details and are additionally featured in lots of different positions, which are seldom sufficiently described in the captions of the training data.