4 ms·

That's incorrect. You can generate the same face with multiple facial expressions by either doing style transfer onto a single face from other faces using the s

by gwern 7y ago

That's incorrect. You can generate the same face with multiple facial expressions by either doing style transfer onto a single face from other faces using the style noise (as demonstrated in the StyleGAN paper & video, and you can see further examples with my anime StyleGAN faces), or you can directly control a face by modifying its latent vector as desired, once you've reverse-engineered the latent vector (as can be done by generating random faces+latents, & feeding the faces into a tagger like DeepDanbooru which knows about facial expressions to do classification/regression). See https://www.gwern.net/Faces#reversing-stylegan-to-control-modify-images https://www.gwern.net/Faces#reversing-stylegan-to-control-mo...

gwern 7y ago
For a recent cool example, http://everyoneishappy.com/portfolio/waifu-synthesis-real-time-generative-anime/ http://everyoneishappy.com/portfolio/waifu-synthesis-real-ti... controls the mouth for sing-along syncing.
TeMPOraL 7y ago
Wow. Thanks for that; I'm very happy to have been wrong about this issue, and I've just learned a lot from reading your Faces page. One issue I notice in your video of interactive face editing is what I'd call "crosstalk" (there's probably a more formal term for this) - you alter a specific slider, and beyond the targeted modification, the entire face, shirt and background changes slightly. Is there any work on eliminating such crosstalk, so one could e.g. stitch together a believable video from such dynamically modified face?
- gwern 7y ago
  My theory is that the latent space isn't perfectly 'disentangled' (there's not really anything in StyleGAN akin to InfoGAN which would force it to be exactly linear/disentangled, it just nicely works out roughly disentangled), so when you train a linear model on the tags+latent and manipulate it linearly, you get slight changes in correlated features. Thus, the slight 'glitchiness'. Not something you'd notice in image samples, but more apparent when you generate interpolations of the 'same' face. If you mess with the style noise only and leave the latent vector alone, the glitchiness should be less, but you also lose a lot of possible control. Training a more powerful nonlinear classifier like a random forests should be able to compensate for the remaining nonlinearity in the latents, I think, but then it's harder to do the actual modification of the latent vector, of course, since you're no longer simply tweaking one variable but many simultaneously to try to hold the other features constant. One could try to brute force it by doing blackbox optimization of the nonlinear classifier... but I haven't seen anyone try that.

by gwern 7y ago

gwern 7y ago

For a recent cool example, http://everyoneishappy.com/portfolio/waifu-synthesis-real-time-generative-anime/ http://everyoneishappy.com/portfolio/waifu-synthesis-real-ti... controls the mouth for sing-along syncing.

TeMPOraL 7y ago

Wow. Thanks for that; I'm very happy to have been wrong about this issue, and I've just learned a lot from reading your Faces page. One issue I notice in your video of interactive face editing is what I'd call "crosstalk" (there's probably a more formal term for this) - you alter a specific slider, and beyond the targeted modification, the entire face, shirt and background changes slightly. Is there any work on eliminating such crosstalk, so one could e.g. stitch together a believable video from such dynamically modified face?

gwern 7y ago
My theory is that the latent space isn't perfectly 'disentangled' (there's not really anything in StyleGAN akin to InfoGAN which would force it to be exactly linear/disentangled, it just nicely works out roughly disentangled), so when you train a linear model on the tags+latent and manipulate it linearly, you get slight changes in correlated features. Thus, the slight 'glitchiness'. Not something you'd notice in image samples, but more apparent when you generate interpolations of the 'same' face. If you mess with the style noise only and leave the latent vector alone, the glitchiness should be less, but you also lose a lot of possible control. Training a more powerful nonlinear classifier like a random forests should be able to compensate for the remaining nonlinearity in the latents, I think, but then it's harder to do the actual modification of the latent vector, of course, since you're no longer simply tweaking one variable but many simultaneously to try to hold the other features constant. One could try to brute force it by doing blackbox optimization of the nonlinear classifier... but I haven't seen anyone try that.