9 ms·

Alias-Free GAN

406 points by nurpax 5y ago

goldemerald 5y ago
After styleGAN-2 came out, I couldn't image what improvements could be made over it. This work is truly impressive. The comparisons are illuminative: StyleGAN2's mapping of texture to specific pixel location looks very similar to poorly implemented video-game textures. Perhaps future GAN improvements could come from tricks used in non-AI graphic development.
- tyingq 5y ago
  >I couldn't image what improvements could be made over it Still has the telltale of mismatched ears and/or earrings. This seems the most reliable way to recognize them. Well, and the nondescript background.
  - mzs 5y ago
    Mismatched reflections across eyes is the dead give-away for me.
  - sbierwagen 5y ago
    Teeth too. Partially covered objects in 3D space have been hard for a GAN to figure out. (See also hands) I wonder what dataset you could even use to tell a GAN about human internals. 3D renders of a skull with various layers removed?
    - cout 5y ago
      I've noticed the same thing with ESRGAN -- teeth are always awful. I'm looking forward to the day when someone figured out how to fix that; I have a few sentimental images taken with a cell phone I would love to see upscaled and cleaned up.
Gimpei 5y ago
Those are some creepy pictures! It's like a photo of the demon inside.
jerf 5y ago
That's starting to be high enough quality that you could start considering using that for some Hollywood-grade special effects. That beach morph stuff is pretty impressive. Faces, perhaps not quite there yet because we are so hyper-focused on those biologically, but you could make one heck of a drug trip scene or a Doctor Strange-esque scene with much less effort with some of those techniques, effort perhaps even getting down to the range of Youtuber videos in the near enough future.
- eru 5y ago
  Compare https://news.ycombinator.com/item?id=27559106 https://news.ycombinator.com/item?id=27559106
  - jerf 5y ago
    First, that's not the same technique and it's not being used for the same purpose. Second, Hollywood doesn't care about that problem. They will take the best application of the technique, and they don't care if they have to apply a few manual touchups on the result. As long as there is one way of using the system to do the sort of thing they showed in the sample, it won't matter to them that they can't embed a full video game into the neural network itself. They only care about the happy path of the tech. Someone's probably already starting the company now to use this in special effects, or putting someone on research in an existing company.
    - eru 5y ago
      > Second, Hollywood doesn't care about that problem. Hmm, I wasn't trying to nay-say anything here. I mostly agree with your original comment. See also how in the Gan Theft Auto they are sort-of getting the light reflection for free without having to explicitly teach the network about that parts of physics.
minimaxir 5y ago
The first two demo videos are interesting examples of using StyleCLIP's global directions to guide an image toward a "smiling face" as noted in that paper with smooth interpolation: https://github.com/orpatashnik/StyleCLIP https://github.com/orpatashnik/StyleCLIP I had ran a few chaotic experiments with StyleCLIP a few months ago which would work very well with smooth interpolation: https://minimaxir.com/2021/04/styleclip/ https://minimaxir.com/2021/04/styleclip/
- Chilinot 5y ago
  That first picture of mark zuckerberg smiling is just straight up cursed. Interesting write up though.
  - Doxin 5y ago
    I audibly went "GAH!" when that scrolled into view. Impressive work.
datameta 5y ago
Wow! The rate of progress is truly stunning. I wonder what Refik Anadol could create with this technique.
benrbray 5y ago
Interesting to that this method makes use of Equivariant Neural Networks. Taco Cohen recently published his PhD thesis [1], which combines a dozen or so papers he authored on the topic. [1]: https://pure.uva.nl/ws/files/60770359/Thesis.pdf https://pure.uva.nl/ws/files/60770359/Thesis.pdf
fogof 5y ago
You can see what they're saying about the fixed in place features with the beards in the first video, but StyleGAN gets the teeth symmetry right whereas this work seems to have trouble with it. Why don't the teeth in the StyleGAN slide around like the beard does?
- minimaxir 5y ago
  That's likely the GANSpace/SeFa part of the manipulation. > In a further test we created two example cinemagraphs that mimic small-scale head movement and facial animation in FFHQ. The geometric head motion was generated as a random latent space walk along hand-picked directions from GANSpace [24] and SeFa [50]. The changes in expression were realized by applying the “global directions” method of StyleCLIP [45], using the prompts “angry face”, “laughing face”, “kissing face”, “sad face”, “singing face”, and “surprised face”. The differences between StyleGAN2 and Alias-Free GAN are again very prominent, with the former displaying jarring sticking of facial hair and skin texture, even under subtle movements
- Geee 5y ago
  In video 9 teeth are sliding.
Imnimo 5y ago
I wonder if you could make the noise inputs work again by using the same process as for the latent code - generate the noise in the frequency domain, and apply the same shift and careful downsampling. If you apply the same shift to the noise as to the latent code, then maybe the whole thing will still be equivariant? In other words, it seems like the problem with the per-pixel noise inputs is that they stay stationary while the latent is shifted, so just shift them also!
ansk 5y ago
This group of researchers consistently demonstrates a degree of empirical rigor that is unmatched across any other ML lab in industry or academia - remarkable empirical results as always, reproducible experiments, open-source and well-engineered codebase, and valuable insights about low-level learning dynamics and high-level emergent artifacts. Applied ML wouldn't have such a bad rap if more researchers held themselves to similar standards.
- sillysaurusx 5y ago
  This isn't true. I do ML every day. You are mistaken. I click the website. I search "model". I see two results. Oh no, that means no download link to model. I go to the github. Maybe model download link is there. I see zero code: https://github.com/NVlabs/alias-free-gan https://github.com/NVlabs/alias-free-gan Zero code. Zero model. You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing damage by saying that this is correct and normal. The thing that's useful to me, first and foremost, is a model. Code alone isn't useful. Code, however, is the recipe to create the model. It might take 400 hours on a V100, and it might not actually result in the model being created, but it slightly helps me. There is no code here. Do you think that the pdf is helpful? Yeah, maybe. But I'm starting to suspect that the pdf is in fact a tech demo for nVidia, not a scientific contribution whose purpose is to be helpful to people like me. Okay? Model first. Code second. Paper third. Every time a tech demo like this comes out, I'd like you to check that those things exist, in that order. If it doesn't, it's not reproducible science. It's a tech demo. I need to write something about this somewhere, because a large number of people seem to be caught in this spell. You're definitely not alone, and I'm sorry for sounding like I was singling you out. I just loaded up the comment section, saw your comment, thought "Oh, awesome!" clicked through, and went "Oh no..."
  - aaron-santos 5y ago
    Thank you for calling this out. It's critically important that people understand the difference between model, code, and paper and what they mean. It's also important that people understand that even if code is provided, it's commercially useless. From the NVAE license as an example[1] > The Work and any derivative works thereof only may be used or intended for use non-commercially. It's a great example of the difference between open source (which it is) and free software which it is not. So we're back to square one where it is probably best to clean-room the implementation from the paper, which is nearly useless to reproduce the model. [1] https://github.com/NVlabs/NVAE/blob/master/LICENSE https://github.com/NVlabs/NVAE/blob/master/LICENSE
ChuckNorris89 5y ago
I expect this work will feed back into removing the aliasing artifacts you sometimes get when using DLSS in games.
Lichtso 5y ago
The previous approaches learned screen-space-textures for different features and a feature mask to compose them. Now it seems to actually learn the topology lines of the human face [0], as 3D artists would learn them [1] when they study anatomy. It also uses quad grids and even places the edge loops and poles in similar places. [0] https://nvlabs-fi-cdn.nvidia.com/_web/alias-free-gan/img/alias-free-gan-teaser-1920x1006.png https://nvlabs-fi-cdn.nvidia.com/_web/alias-free-gan/img/ali... [1] https://i.pinimg.com/originals/6b/9a/0c/6b9a0c2d108b2be75bf72da8460cceb9.jpg https://i.pinimg.com/originals/6b/9a/0c/6b9a0c2d108b2be75bf7...
- eru 5y ago
  Yes. It's interesting that imposing what are essentially 2d invariance constraints leads the network to learn what we regard as 3D concepts.
  - pvillano 5y ago
    There are some interesting 2d things our eyes do for 3d. If something is on the ground, half is above the horizon and half is below. Parallax is a 2d phenomenon.
isoprophlex 5y ago
If ReLU-introduced high frequency components are indeed the culprit, won't using "softened" ReLU (without discontinuity in the derivative at 0) everywhere solve the problem, too?
Bjartr 5y ago
That beach interpolation is begging for a music video
russdpale 5y ago
Great work!
evo 5y ago
I wonder if there are learnings from this that could be transposed into the 1-D domain for audio; as far as I know, aliasing is a frequent challenge when using deep learning methods for audio (e.g. simulating non-linear circuits for guitar amps).
l_d_s 5y ago
The internal representations (Video 8) look suspiciously like The Lawnmower Man ...
ipunchghosts 5y ago
How does this differ from Richard zhang's work?
forgotpwd16 5y ago
Why weren't the same pictures used for StyleGAN2 and Alias-Free GAN?
- dannyw 5y ago
  Because the latent space to picture pipeline is considerably different. There's no weight to output compatibility. If you ask styleGAN to generate a specific image, that's possible, but you are no longer looking at how well these models generate images.
eru 5y ago
Their examples look much better in some objective sense. Especially if you want to create something that looks realistic in animation. But I do appreciate the artefacts of StyleGAN2 as an artistic choice, too.

406 points by nurpax 5y ago

After styleGAN-2 came out, I couldn't image what improvements could be made over it. This work is truly impressive. The comparisons are illuminative: StyleGAN2's mapping of texture to specific pixel location looks very similar to poorly implemented video-game textures. Perhaps future GAN improvements could come from tricks used in non-AI graphic development.

tyingq 5y ago
>I couldn't image what improvements could be made over it Still has the telltale of mismatched ears and/or earrings. This seems the most reliable way to recognize them. Well, and the nondescript background.
- mzs 5y ago
  Mismatched reflections across eyes is the dead give-away for me.
- sbierwagen 5y ago
  Teeth too. Partially covered objects in 3D space have been hard for a GAN to figure out. (See also hands) I wonder what dataset you could even use to tell a GAN about human internals. 3D renders of a skull with various layers removed?
  - cout 5y ago
    I've noticed the same thing with ESRGAN -- teeth are always awful. I'm looking forward to the day when someone figured out how to fix that; I have a few sentimental images taken with a cell phone I would love to see upscaled and cleaned up.

Gimpei 5y ago

Those are some creepy pictures! It's like a photo of the demon inside.

jerf 5y ago

That's starting to be high enough quality that you could start considering using that for some Hollywood-grade special effects. That beach morph stuff is pretty impressive. Faces, perhaps not quite there yet because we are so hyper-focused on those biologically, but you could make one heck of a drug trip scene or a Doctor Strange-esque scene with much less effort with some of those techniques, effort perhaps even getting down to the range of Youtuber videos in the near enough future.

eru 5y ago
Compare https://news.ycombinator.com/item?id=27559106 https://news.ycombinator.com/item?id=27559106
- jerf 5y ago
  First, that's not the same technique and it's not being used for the same purpose. Second, Hollywood doesn't care about that problem. They will take the best application of the technique, and they don't care if they have to apply a few manual touchups on the result. As long as there is one way of using the system to do the sort of thing they showed in the sample, it won't matter to them that they can't embed a full video game into the neural network itself. They only care about the happy path of the tech. Someone's probably already starting the company now to use this in special effects, or putting someone on research in an existing company.
  - eru 5y ago
    > Second, Hollywood doesn't care about that problem. Hmm, I wasn't trying to nay-say anything here. I mostly agree with your original comment. See also how in the Gan Theft Auto they are sort-of getting the light reflection for free without having to explicitly teach the network about that parts of physics.

minimaxir 5y ago

The first two demo videos are interesting examples of using StyleCLIP's global directions to guide an image toward a "smiling face" as noted in that paper with smooth interpolation: https://github.com/orpatashnik/StyleCLIP https://github.com/orpatashnik/StyleCLIP I had ran a few chaotic experiments with StyleCLIP a few months ago which would work very well with smooth interpolation: https://minimaxir.com/2021/04/styleclip/ https://minimaxir.com/2021/04/styleclip/

Chilinot 5y ago
That first picture of mark zuckerberg smiling is just straight up cursed. Interesting write up though.
- Doxin 5y ago
  I audibly went "GAH!" when that scrolled into view. Impressive work.

datameta 5y ago

Wow! The rate of progress is truly stunning. I wonder what Refik Anadol could create with this technique.

benrbray 5y ago

Interesting to that this method makes use of Equivariant Neural Networks. Taco Cohen recently published his PhD thesis [1], which combines a dozen or so papers he authored on the topic. [1]: https://pure.uva.nl/ws/files/60770359/Thesis.pdf https://pure.uva.nl/ws/files/60770359/Thesis.pdf

fogof 5y ago

You can see what they're saying about the fixed in place features with the beards in the first video, but StyleGAN gets the teeth symmetry right whereas this work seems to have trouble with it. Why don't the teeth in the StyleGAN slide around like the beard does?

minimaxir 5y ago
That's likely the GANSpace/SeFa part of the manipulation. > In a further test we created two example cinemagraphs that mimic small-scale head movement and facial animation in FFHQ. The geometric head motion was generated as a random latent space walk along hand-picked directions from GANSpace [24] and SeFa [50]. The changes in expression were realized by applying the “global directions” method of StyleCLIP [45], using the prompts “angry face”, “laughing face”, “kissing face”, “sad face”, “singing face”, and “surprised face”. The differences between StyleGAN2 and Alias-Free GAN are again very prominent, with the former displaying jarring sticking of facial hair and skin texture, even under subtle movements
Geee 5y ago
In video 9 teeth are sliding.

Imnimo 5y ago

I wonder if you could make the noise inputs work again by using the same process as for the latent code - generate the noise in the frequency domain, and apply the same shift and careful downsampling. If you apply the same shift to the noise as to the latent code, then maybe the whole thing will still be equivariant? In other words, it seems like the problem with the per-pixel noise inputs is that they stay stationary while the latent is shifted, so just shift them also!

ansk 5y ago

This group of researchers consistently demonstrates a degree of empirical rigor that is unmatched across any other ML lab in industry or academia - remarkable empirical results as always, reproducible experiments, open-source and well-engineered codebase, and valuable insights about low-level learning dynamics and high-level emergent artifacts. Applied ML wouldn't have such a bad rap if more researchers held themselves to similar standards.

sillysaurusx 5y ago
This isn't true. I do ML every day. You are mistaken. I click the website. I search "model". I see two results. Oh no, that means no download link to model. I go to the github. Maybe model download link is there. I see zero code: https://github.com/NVlabs/alias-free-gan https://github.com/NVlabs/alias-free-gan Zero code. Zero model. You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing damage by saying that this is correct and normal. The thing that's useful to me, first and foremost, is a model. Code alone isn't useful. Code, however, is the recipe to create the model. It might take 400 hours on a V100, and it might not actually result in the model being created, but it slightly helps me. There is no code here. Do you think that the pdf is helpful? Yeah, maybe. But I'm starting to suspect that the pdf is in fact a tech demo for nVidia, not a scientific contribution whose purpose is to be helpful to people like me. Okay? Model first. Code second. Paper third. Every time a tech demo like this comes out, I'd like you to check that those things exist, in that order. If it doesn't, it's not reproducible science. It's a tech demo. I need to write something about this somewhere, because a large number of people seem to be caught in this spell. You're definitely not alone, and I'm sorry for sounding like I was singling you out. I just loaded up the comment section, saw your comment, thought "Oh, awesome!" clicked through, and went "Oh no..."
- aaron-santos 5y ago
  Thank you for calling this out. It's critically important that people understand the difference between model, code, and paper and what they mean. It's also important that people understand that even if code is provided, it's commercially useless. From the NVAE license as an example[1] > The Work and any derivative works thereof only may be used or intended for use non-commercially. It's a great example of the difference between open source (which it is) and free software which it is not. So we're back to square one where it is probably best to clean-room the implementation from the paper, which is nearly useless to reproduce the model. [1] https://github.com/NVlabs/NVAE/blob/master/LICENSE https://github.com/NVlabs/NVAE/blob/master/LICENSE

ChuckNorris89 5y ago

I expect this work will feed back into removing the aliasing artifacts you sometimes get when using DLSS in games.

Lichtso 5y ago

The previous approaches learned screen-space-textures for different features and a feature mask to compose them. Now it seems to actually learn the topology lines of the human face [0], as 3D artists would learn them [1] when they study anatomy. It also uses quad grids and even places the edge loops and poles in similar places. [0] https://nvlabs-fi-cdn.nvidia.com/_web/alias-free-gan/img/alias-free-gan-teaser-1920x1006.png https://nvlabs-fi-cdn.nvidia.com/_web/alias-free-gan/img/ali... [1] https://i.pinimg.com/originals/6b/9a/0c/6b9a0c2d108b2be75bf72da8460cceb9.jpg https://i.pinimg.com/originals/6b/9a/0c/6b9a0c2d108b2be75bf7...

eru 5y ago
Yes. It's interesting that imposing what are essentially 2d invariance constraints leads the network to learn what we regard as 3D concepts.
- pvillano 5y ago
  There are some interesting 2d things our eyes do for 3d. If something is on the ground, half is above the horizon and half is below. Parallax is a 2d phenomenon.

isoprophlex 5y ago

If ReLU-introduced high frequency components are indeed the culprit, won't using "softened" ReLU (without discontinuity in the derivative at 0) everywhere solve the problem, too?

Bjartr 5y ago

That beach interpolation is begging for a music video

russdpale 5y ago

Great work!

evo 5y ago

I wonder if there are learnings from this that could be transposed into the 1-D domain for audio; as far as I know, aliasing is a frequent challenge when using deep learning methods for audio (e.g. simulating non-linear circuits for guitar amps).

l_d_s 5y ago

The internal representations (Video 8) look suspiciously like The Lawnmower Man ...

ipunchghosts 5y ago

How does this differ from Richard zhang's work?

forgotpwd16 5y ago

Why weren't the same pictures used for StyleGAN2 and Alias-Free GAN?

dannyw 5y ago
Because the latent space to picture pipeline is considerably different. There's no weight to output compatibility. If you ask styleGAN to generate a specific image, that's possible, but you are no longer looking at how well these models generate images.

eru 5y ago

Their examples look much better in some objective sense. Especially if you want to create something that looks realistic in animation. But I do appreciate the artefacts of StyleGAN2 as an artistic choice, too.