6 ms·

StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis

109 points by lnyan 5y ago

Allstar 5y ago
The fact that this was in part developed with the help of Facebook AI Research is kind of unsettling. That said, the results are actually very impressive.
- throwoutway 5y ago
  Didn’t Facebook just get sued for illegally harvesting faces & tags of people? If derivatives of that data make it into research like this, where does that stand from a legal perspective?
- blurbleblurble 5y ago
  Kinda puts a spin on the "face" in "facebook"
  - thedorkknight 5y ago
    It literally puts a spin on it, actually!
- axg11 5y ago
  It's only unsettling if you are completely unfamiliar with the machine learning field. FAIR publishes research in many ML/AI fields. They're responsible for plenty of advances and contributions.
  - truemotive 5y ago
    I’m not sure that’s what was meant by unsettling…
motohagiography 5y ago
While watching it, I wondered whether the "this x does not exist," GAN sites also implies a subset of "this x must exist" and if looking at it that way could yield discoveries in other areas. e.g. This X must exist, where x was a solar system, protein, wave function, chemical substance, cell, transcendental number, etc. Naive view, but when you're extapolating from a field of arbitrary 2D data, it seems there are a lot of ladders you could climb with that.
axg11 5y ago
I see there are a lot of questions/doubts in the comments about the usefulness of such research. This type of research is the precursor to gaining the ability to generative photorealistic 3D worlds. What is that useful for? Anywhere we currently generate 3D environments, including movies, games, VR/AR.
kingcharles 5y ago
GANs are getting creepier by the day.
- nicoco 5y ago
  Do you mean their potential (mis)use is creepy, or are you genuinely scared by their "abilities"?
  - nlehuen 5y ago
    This whole research domain is confusing to me. I do get that it's a hard and cool problem to solve and I do see why researcher could work on it on those grounds. But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.". I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already). So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show). What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?
    - s0teri0s 5y ago
      I guess the non-creepy application I'm seeing for the future of this research is the seamless conversion of a 2D image into 3D for when 3D projectors become more commonplace and perhaps overtake the ubiquity of flatscreens on our devices. And I am not sure, but it might also be applicable to the video-cam issue where we want to make it look as though the person we're having a video conference with is looking at us instead of the screen image of us and vice-versa.
    - antegamisou 5y ago
fxtentacle 5y ago
Cool idea :) 1. They generate a low-resolution position map using NeRF => Good camera and 3D control 2. They then upsample that with StyleGAN2 => Good features in the final 2D image. So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.
egnehots 5y ago
that's weird that they mostly ignore the comparison to stylegan3: https://nvlabs.github.io/stylegan3/ https://nvlabs.github.io/stylegan3/ it's not a nerf based solution but since the goal is high resolution image syntesis..
petesergeant 5y ago
subtitle: an amazing trip through the uncanny valley
harry-wood 5y ago
The twinkling children's TV style background music in that video is not making it any less creepy.
birdyrooster 5y ago
Interesting that the reflections in each eye vary so much between each other.
- Mandelmus 5y ago
  This might fix that: https://dorverbin.github.io/refnerf/ https://dorverbin.github.io/refnerf/

109 points by lnyan 5y ago

The fact that this was in part developed with the help of Facebook AI Research is kind of unsettling. That said, the results are actually very impressive.

throwoutway 5y ago
Didn’t Facebook just get sued for illegally harvesting faces & tags of people? If derivatives of that data make it into research like this, where does that stand from a legal perspective?
blurbleblurble 5y ago
Kinda puts a spin on the "face" in "facebook"
- thedorkknight 5y ago
  It literally puts a spin on it, actually!
axg11 5y ago
It's only unsettling if you are completely unfamiliar with the machine learning field. FAIR publishes research in many ML/AI fields. They're responsible for plenty of advances and contributions.
- truemotive 5y ago
  I’m not sure that’s what was meant by unsettling…

motohagiography 5y ago

While watching it, I wondered whether the "this x does not exist," GAN sites also implies a subset of "this x must exist" and if looking at it that way could yield discoveries in other areas. e.g. This X must exist, where x was a solar system, protein, wave function, chemical substance, cell, transcendental number, etc. Naive view, but when you're extapolating from a field of arbitrary 2D data, it seems there are a lot of ladders you could climb with that.

axg11 5y ago

I see there are a lot of questions/doubts in the comments about the usefulness of such research. This type of research is the precursor to gaining the ability to generative photorealistic 3D worlds. What is that useful for? Anywhere we currently generate 3D environments, including movies, games, VR/AR.

kingcharles 5y ago

GANs are getting creepier by the day.

nicoco 5y ago
Do you mean their potential (mis)use is creepy, or are you genuinely scared by their "abilities"?
- nlehuen 5y ago
  This whole research domain is confusing to me. I do get that it's a hard and cool problem to solve and I do see why researcher could work on it on those grounds. But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.". I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already). So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show). What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?
  - s0teri0s 5y ago
    I guess the non-creepy application I'm seeing for the future of this research is the seamless conversion of a 2D image into 3D for when 3D projectors become more commonplace and perhaps overtake the ubiquity of flatscreens on our devices. And I am not sure, but it might also be applicable to the video-cam issue where we want to make it look as though the person we're having a video conference with is looking at us instead of the screen image of us and vice-versa.
  - antegamisou 5y ago

fxtentacle 5y ago

Cool idea :) 1. They generate a low-resolution position map using NeRF => Good camera and 3D control 2. They then upsample that with StyleGAN2 => Good features in the final 2D image. So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.

egnehots 5y ago

that's weird that they mostly ignore the comparison to stylegan3: https://nvlabs.github.io/stylegan3/ https://nvlabs.github.io/stylegan3/ it's not a nerf based solution but since the goal is high resolution image syntesis..

petesergeant 5y ago

subtitle: an amazing trip through the uncanny valley

harry-wood 5y ago

The twinkling children's TV style background music in that video is not making it any less creepy.

birdyrooster 5y ago

Interesting that the reflections in each eye vary so much between each other.

Mandelmus 5y ago
This might fix that: https://dorverbin.github.io/refnerf/ https://dorverbin.github.io/refnerf/