6 ms·

Neural Actor: Neural Free-View Synthesis of Human Actors with Pose Control

143 points by Hard_Space 5y ago

whitten 5y ago
So this method seems to create an inner core "skeleton" that is then used with various shells. How is this method compared to a commercial product such as Poser ?
- qayxc 5y ago
  Does Poser work with photos and videos as inputs? If not, there's your answer.
  - bsenftner 5y ago
    No.
- gedy 5y ago
  You don't need to create or capture a mesh. This is synthesizing one from a few 2d images, with impressively accurate results.
- wokwokwok 5y ago
  Poser requires an existing rigged, textured mesh. Poser requires that you manually animate the skeleton. This approach is more like a mixture of mocap/photometry; given an input video (a) and (b), synthesise a fully textured model from (a) and apply the motion from (b) on it. Of course, the output of this is horribly bad compared to state of the art 3d models; the point is that the inputs are trivial and it shows that NeRF are increasingly plausible as a solution for real use cases. Of course... this is all a bit 'proof of concept stuff'. > To further evaluate our method on a wider variety of body poses and more challenging textured clothing, we captured a new multi-view human performance corpus with 79 − 86 cameras at a resolution of 1285 × 940. It contains four sequences, 𝑁1-𝑁4, and each has 12,000−16,000 frames for training and around 8, 000 frames for testing These captured images are also, notably, green screen, although not with the traditional mocap markers. So, no, you're not going to get a 3d animation from a video off your phone anytime even remotely soon. ...but, it's still pretty darn awesome, and they're going to release the datasets for others to play with.
Reason077 5y ago
"Stantard deviation"? Typo?
montroser 5y ago
This is incredible. Interesting how combining the neural net with 3d mesh and texture map provides the best results of all the approaches they tried. I am both excited and terrified for what this stuff may yield in our future realities.
hliyan 5y ago
Another 10 years of progress, better facial expression control and voice synthesis, I suspect highly paid actors will be a thing of the past.
- qayxc 5y ago
  This has been prophesised before (last in the early 2000s) and didn't come to pass. I wouldn't expect that to change given this technology either. There's a market that would benefit greatly from this, though: marketing and training videos as well as stock-footage. Some companies (Samsung comes to mind) already use synthetic avatars to represent their products. Artificial influencers and the like will also take the place of real humans. Film actors, though, not so much. There's more to a successful actor than just appearing in film (who'd sit at meet & greets, autograph signings or attend shows?) so there's little reason to replace living actors. Deceased ones, though...
  - makapuf 5y ago
    The tendency is even reverse with highly paid actors being hired for only the voice acting featuring in computer-animated movies (where great voice-only performers would be cheaper)
    - refactor_master 5y ago
      Humans basically want to belong to a tribe, which is why there’s always a certain equilibrium with relatively few “famous” people on one side. If AI and the Internet made everyone exceptional, nobody would be exceptional, and we’d just find random idols to gather around anyway (sort of what we already do). Sure, you could use AI to act, or you could follow your most basic gut instinct and slap a famous-for-being-famous person on it and call it a day. Because it’s going to work.
      - CyberDildonics 5y ago
        This is nonsense. First, actors are not famous for being famous, they are famous because people recognize them and eventually learn their names. Second, the name recognition is why they get cast as voices. Their name can then be used to promote the movie and the actors themselves can go out and promote the movie.
spywaregorilla 5y ago
Interesting. My first observation is that the rendered people all have really tight clothing on. The model guesses at some creases, but it probably wouldn't guesstimate cloth physics very well. I would tend to think you would have much better results synthesizing a mesh with a NN and getting a game engine or something to do this part with the animation.
- ridaj 5y ago
  That's especially visible when comparing with the "driving person", who is floating in a loose open trench-coat that flits about as they dance
iandanforth 5y ago
This page causes multiple mp4 download prompts to open in Firefox (v89.0, OSX) for me. Anyone else seeing this?
- philote 5y ago
  Not for me (v88.0.1, OSX)
- Hard_Space 5y ago
  No, not for me. I think you'll need to review file association actions in Firefox prefs. I have to do this regularly for PDF.
- jkingsman 5y ago
  I'm on Chrome 90.0.4430.212 on Catalina and I had multiple videos with narration all start on page-open. Had to mute the page because I couldn't figure out which one was talking.
drcode 5y ago
I assume we'll see some optimized version of this for NPCs in video games soon.
kossTKR 5y ago
Love the dancing ha! Also is this ready to be used on a small project, knowing very little og NN’s? I just need s bit of motion tracking to be used on various figures on a webgl website, have been looking at the iphone depth sensor, the kinect and Mixamo.
indiantinker 5y ago
Anybody can dance now! yay!
fareesh 5y ago
TikTok filter - now you don't need to dance, just use the filter
TimPC 5y ago
I’m very glad they used a hyphen in the title as it completely changes the expected meaning.
grawprog 5y ago
The results remind me of high quality rotoscoped characters. Like if the original Mortal Kombat got an HD remaster or something. They look good but not real. If they do start replacing actors with these, everything's gonna look like a Ralph Bakshi movie.
m3kw9 5y ago
How is this different from putting a skin on the real actor?
Abishek_Muthian 5y ago
Soon those who have motor control issues or other problems would be able to just feed the script and get their virtual self on video with all the movements which isn't possible in reality. Which might be necessary, Considering even a LinkedIn profile requires some TikToking now in the name of 'Introduce yourself in a 30 second video'.
TchoBeer 5y ago
My first thought is that this will make for some pretty good memes
l1n 5y ago
I really really wish that there was a norm of uploading code at the same time as the paper. The number of "code coming soon!" buttons on these sorts of sites (oftentimes several years in the past) is far too high. If you can't play around with the model, how can you tell the level of cherry-picking that the method has?

143 points by Hard_Space 5y ago

So this method seems to create an inner core "skeleton" that is then used with various shells. How is this method compared to a commercial product such as Poser ?

qayxc 5y ago
Does Poser work with photos and videos as inputs? If not, there's your answer.
- bsenftner 5y ago
  No.
gedy 5y ago
You don't need to create or capture a mesh. This is synthesizing one from a few 2d images, with impressively accurate results.
wokwokwok 5y ago
Poser requires an existing rigged, textured mesh. Poser requires that you manually animate the skeleton. This approach is more like a mixture of mocap/photometry; given an input video (a) and (b), synthesise a fully textured model from (a) and apply the motion from (b) on it. Of course, the output of this is horribly bad compared to state of the art 3d models; the point is that the inputs are trivial and it shows that NeRF are increasingly plausible as a solution for real use cases. Of course... this is all a bit 'proof of concept stuff'. > To further evaluate our method on a wider variety of body poses and more challenging textured clothing, we captured a new multi-view human performance corpus with 79 − 86 cameras at a resolution of 1285 × 940. It contains four sequences, 𝑁1-𝑁4, and each has 12,000−16,000 frames for training and around 8, 000 frames for testing These captured images are also, notably, green screen, although not with the traditional mocap markers. So, no, you're not going to get a 3d animation from a video off your phone anytime even remotely soon. ...but, it's still pretty darn awesome, and they're going to release the datasets for others to play with.

Reason077 5y ago

"Stantard deviation"? Typo?

montroser 5y ago

This is incredible. Interesting how combining the neural net with 3d mesh and texture map provides the best results of all the approaches they tried. I am both excited and terrified for what this stuff may yield in our future realities.

hliyan 5y ago

Another 10 years of progress, better facial expression control and voice synthesis, I suspect highly paid actors will be a thing of the past.

qayxc 5y ago
This has been prophesised before (last in the early 2000s) and didn't come to pass. I wouldn't expect that to change given this technology either. There's a market that would benefit greatly from this, though: marketing and training videos as well as stock-footage. Some companies (Samsung comes to mind) already use synthetic avatars to represent their products. Artificial influencers and the like will also take the place of real humans. Film actors, though, not so much. There's more to a successful actor than just appearing in film (who'd sit at meet & greets, autograph signings or attend shows?) so there's little reason to replace living actors. Deceased ones, though...
- makapuf 5y ago
  The tendency is even reverse with highly paid actors being hired for only the voice acting featuring in computer-animated movies (where great voice-only performers would be cheaper)
  - refactor_master 5y ago
    Humans basically want to belong to a tribe, which is why there’s always a certain equilibrium with relatively few “famous” people on one side. If AI and the Internet made everyone exceptional, nobody would be exceptional, and we’d just find random idols to gather around anyway (sort of what we already do). Sure, you could use AI to act, or you could follow your most basic gut instinct and slap a famous-for-being-famous person on it and call it a day. Because it’s going to work.
    - CyberDildonics 5y ago
      This is nonsense. First, actors are not famous for being famous, they are famous because people recognize them and eventually learn their names. Second, the name recognition is why they get cast as voices. Their name can then be used to promote the movie and the actors themselves can go out and promote the movie.

spywaregorilla 5y ago

Interesting. My first observation is that the rendered people all have really tight clothing on. The model guesses at some creases, but it probably wouldn't guesstimate cloth physics very well. I would tend to think you would have much better results synthesizing a mesh with a NN and getting a game engine or something to do this part with the animation.

ridaj 5y ago
That's especially visible when comparing with the "driving person", who is floating in a loose open trench-coat that flits about as they dance

iandanforth 5y ago

This page causes multiple mp4 download prompts to open in Firefox (v89.0, OSX) for me. Anyone else seeing this?

philote 5y ago
Not for me (v88.0.1, OSX)
Hard_Space 5y ago
No, not for me. I think you'll need to review file association actions in Firefox prefs. I have to do this regularly for PDF.
jkingsman 5y ago
I'm on Chrome 90.0.4430.212 on Catalina and I had multiple videos with narration all start on page-open. Had to mute the page because I couldn't figure out which one was talking.

drcode 5y ago

I assume we'll see some optimized version of this for NPCs in video games soon.

kossTKR 5y ago

Love the dancing ha! Also is this ready to be used on a small project, knowing very little og NN’s? I just need s bit of motion tracking to be used on various figures on a webgl website, have been looking at the iphone depth sensor, the kinect and Mixamo.

indiantinker 5y ago

Anybody can dance now! yay!

fareesh 5y ago

TikTok filter - now you don't need to dance, just use the filter

TimPC 5y ago

I’m very glad they used a hyphen in the title as it completely changes the expected meaning.

grawprog 5y ago

The results remind me of high quality rotoscoped characters. Like if the original Mortal Kombat got an HD remaster or something. They look good but not real. If they do start replacing actors with these, everything's gonna look like a Ralph Bakshi movie.

m3kw9 5y ago

How is this different from putting a skin on the real actor?

Abishek_Muthian 5y ago

Soon those who have motor control issues or other problems would be able to just feed the script and get their virtual self on video with all the movements which isn't possible in reality. Which might be necessary, Considering even a LinkedIn profile requires some TikToking now in the name of 'Introduce yourself in a 30 second video'.

TchoBeer 5y ago

My first thought is that this will make for some pretty good memes

l1n 5y ago

I really really wish that there was a norm of uploading code at the same time as the paper. The number of "code coming soon!" buttons on these sorts of sites (oftentimes several years in the past) is far too high. If you can't play around with the model, how can you tell the level of cherry-picking that the method has?