16 ms·
Disney Combines CGI with Neural Rendering to Tackle the ‘Uncanny Valley’
- deleted 5y ago[deleted]
- wiz21c 5y agoStrange, the more un-uncanny the more uncanny it feels. For example, the way the eyes are oriented seems a bit off when compared to the posture of the muscles around the eyes: the eyes seems to look in one direction while these muscles seem to look to another. Is it just me ?
- tdrdt 5y agoI once talked to a painter who painted stunning portraits. He said that it took him less than a day to paint the portrait but then days to make the eyes and mouth believable.
- skydhash 5y agoI confirmed. Even when doing sketch, the key thing to get right is the eyes.
- 6nf 5y agoI think that means we're still on the far side of the uncanny valley
- usrusr 5y agoI agree, just look at the chin of the blue background face at 7:00. They've just dug a deeper valley. But it works so well for stills that I could imagine it taking over fast fashion catalog shots: the nameless manufacturer/designer back-end provides basic pictures (with a lighting reference ball conveniently in frame), front-end brand/store chain/online shop composes in a cgi face from their brand identity library if they don't feel like keeping the of supplied.
- amelius 5y agoWith increased use of fillers/botox/surgery in the cosmetics industry, the real world and the uncanny valley will converge somewhere in the middle.
- wigster 5y agotoo bad the video doesn't work. is it just me, or taken down?
- Flankk 5y agoSo many women have Botox nowadays. I don't get the feeling of dread I used to from these dead plasticine faces.
- DalasNoin 5y agoI assume the end goal of this is rendering photo-realistic movies of humans without filming actors. Neural voice generation is also quite good at this point. Quite interesting where this could lead to.
- path411 5y agoI think this tech is still only for faces mostly. Aren't any cgi'd actor have someone else playing the role and their face edited?
- devoutsalsa 5y agoFor actors that seem to play the same character in every movie, it’d be kind of funny to create a stock footage of actors, and just use that to make movies. Tom Cruise running! The Rock punching someone through a wall! And so on.
- bemmu 5y agoIn 2030 I'll type in "movie about XYZ", GPT-7 comes up with the script, it crunches in the Disney+ cloud for a few hours, and later I sign on to watch the full generated movie?
- ricardobayes 5y agoprobably not going to be mainstream for feature length movies, but could be a good tool for prototyping movie ideas and let target groups watch the demo reel very, very early on in the concept phase.
- suriyaG 5y agoWhy use a target group? when you can run a regression model on what hits the nerve for people, using data that's being collected now. :) Then it'll be ML models creating movies for other models to be validated and et voila, you get a super hit movie.
- ricardobayes 5y ago
- onion2k 5y agoThe examples have a distinct "police mugshot" look to them.
- GekkePrutser 5y agoLol so true
- ggm 5y agoI think static image canny is getting good. "This(x)doesn't exist.com" proves a lot of realism is down the rabbit hole. Those faces are believable. But I think moving image uncanny Valley is alive and well. Probably I'm setting myself up to be proved wrong.
- toxicFork 5y agoSomething feels really uncanny with them. Especially the nose.
- OscarCunningham 5y agoI wonder if in the future a new face will be created for every character in film or TV. New generations will think it strange that we had characters in different shows looking exactly the same because they were played by the same actor. And they'll never have to kill off a character because the actor quit.
- toxik 5y agoThat would be too bad IMO, even with animated comics like Archer, I really enjoy this kind voice cameos and just recognizing voices from other VO work or even reality - e.g. the characters Malory Archer and Len Trexler.
- londons_explore 5y agoWhen compute becomes cheap, they may do that for every viewer of every show. For example, game of thrones might have it's whole cast replaced with people like locals from your hometown. None of them will be using the same brand of car as you - they'll all use whichever brand wants your custom most.
- lordnacho 5y agoThe story itself will be made to suit you as well. They already do this with the news, so why not with fantasy universes? If you're the cynical type who things large orgs are all messed up, you get a story where the protagonist looks like you and gets stuck in the same BS that you got stuck in.
- Lio 5y ago...I guess pushing that idea further, they could also replace all the product placement after a few years. For a quality show like The Sopranos, where people are still rewatching it and new generations discovering it that could be a good new source of revenue. Change all the vintage Fords to similar vintage GMs. Change Coke to Pepsi. The novelty of this happening the first time would probably get people watching by itself.
- 5y ago
- ricardobayes 5y agoAll this work and probably going to bastardized by Big Facial Recognition. I would argue we don't need better facial recognition, it has no legit use cases. We are one freak deepfake incident away from this idea being mainstream.
- mikhailbolton 5y agoIt will be interesting to see if the better results they are getting is from being able to generate their own superior training data (ground truth)
- 123pie123 5y agothey're still a long way off making it look realistic imho Video at the bottom of the page or here https://www.youtube.com/watch?v=k-RKSGbWLng https://www.youtube.com/watch?v=k-RKSGbWLng animation starts around 6:11
- lhenault 5y agoTo be fair they've been using StyleGAN2 and could probably get much better animations with alias free GAN (aka StyleGAN3). Watch the videos there : https://nvlabs.github.io/stylegan3/ https://nvlabs.github.io/stylegan3/
- warning26 5y agoLooks like the video was taken down -- weird.
- 123pie123 5y agoit's now on https://www.youtube.com/watch?v=TwpLqTmvqVk&t=1s https://www.youtube.com/watch?v=TwpLqTmvqVk&t=1s
- Intermernet 5y agoNote to people making videos like this. PLEASE pay some attention to audio recording and mouth noise. I had so much trouble paying attention to the subject matter due to the terrible recording of the voiceover. It sounds like a chocolate ad for squids. I'm very sensitive to this as I've done a lot of voice recording work, but this is an extreme example if what not to do. A combination of a simple pop shield, some post-proc eq and muting, and drinking a glass of water before recording would have mostly eliminated the problems. I know this is off-topic, but this is arguably as important as getting the visuals right.
- RicoElectrico 5y agoPeople often argue that bad video with good audio is much more palatable than good video with bad audio.
- Gymkana 5y agoI used to work for our state broadcaster. It was well known within the industry that if audio channels went down, or were wrong you'd get more complaints than if video quality dropped.
- klondike_klive 5y agoYou can close your eyes but not your ears.
- micromacrofoot 5y agoAs a species we communicate primarily via sound, typically visual cues are secondary.
- mnw21cam 5y agoLast presentation I recorded for a conference, I just started the voice recorder on my phone and held it near my mouth. The sound quality was very surprisingly good. I could do that because I wasn't also videoing myself, so I didn't have to synchronise them up afterwards. Anything is better than the microphone built into the average laptop.
- Errancer 5y agoSlightly off topic but there is an interesting argument to be made against the existence of 'Uncanny Valley' – https://youtu.be/LKJBND_IRdI https://youtu.be/LKJBND_IRdI.
- mejutoco 5y agoA lot of the examples look creepy. In my opinion, it is the evil/serious/focused eyes, which do not correspond to the big smile. I believe this tension between contrasting expressions seems dishonest.
- deleted 5y ago[deleted]
- Jeff_Brown 5y agoWhy do some of the meshes seem to have extra skin outside of the face, to the sides of the jaw?
- londons_explore 5y agoThis seems like only the first step of many to make something usable in a film... Like what about handling side views of faces? How do you remove the background and blend it into the actual scene? What when the character walks through a shadow and if falls unevenly over part of the face? How about characters interacting (kissing etc)? What about actions like 'putting on lipstick'?
- dwighttk 5y agoThe beginning of article seems to hint at improving on Rogue One’s results, but it seems from the rest of the article and the video they are just trying to automate the inpainting, which gives worse results than Rogue One but a lot easier and less expensive. Like this is a step backwards into the uncanny valley. The “improvement” is not having to pay artists to manually paint hair, eyes, and teeth.
- mromanuk 5y agoYes, I was hoping to see some progress out of the uncanny valley. But this got even deeper in the valley. My 7 year old, was next to me, and she got scared of all these
- protoman3000 5y agoIn the sphere of music we observe that autotune impacted the voice of new generations of singers, making them align more towards the fake/uncanny valley, instead of autotune becoming better to get out of the uncanny valley. This effect is hypothesized to come from the abundance of autotuned voice exposure new singers impress and train their ears with starting from early age. This generates a self feeding loop which on a societal level shifts the location of the uncanny valley. Eventually, to the vast majority of people, old not-autotuned music will sound uncanny or off. Adam Neely made a interesting episode about this: https://youtu.be/yxX2u8iggYI https://youtu.be/yxX2u8iggYI (around 5:00) I wonder if we will observe something similar in acting and social interaction, where more and more bad or CGI acting gets corrected by uncanny valley methods, which new generations get on mass exposed with and thus change their own habits and what it means for facial expressions to be uncanny. There is an argument to be made that due to economic reasons, eventually it will not pay off to invest into reaching 100% out of uncanny valley but rather a “good enough” implementation will gain mass traction and thus change perception of what the uncanny valley is. Old movies will be the uncanny valley.
- skerit 5y agoSo people who are actually talented singers will sound weird in the future? Now that's dystopian. I honestly hear auto-tune pretty quick. It bothers me that other people don't seem to notice it so easily.
- siltpotato 5y agoMaybe not weird so much as not standard. For example, guitarists with good dynamics-control sound different than guitarists of arbitrary dynamics-control quality with a compressor/limiter at the front of their chain.
- Gravityloss 5y agoI think it's had the opposite effect - that people listen to old music a lot more. Even young kids know a lot of eighties music. New music has something uncanny about it and it's tiring to listen to. I think it's related to how it's mixed (for example guitars have a lot of bass) or how the tempo is quantized too.
- hawski 5y agoThose faces with dead eyes look like paintings when they start to talk on movies. They look real statically in comparison to a low quality photos.
- joshspankit 5y agoCan someone tell me why almost all of the “realistic” 3D rendering is about superficial skin deformation? Hear me out: - Train an AI to determine fascial muscle activation from mocap data - Build your animation pipeline to use muscle activation to drive model deformation - Take the actor’s mocap acting and have the system convert their muscle movements to the 3d model’s muscle movements - Tackle the uncanny valley by refining how well we implement the bone/skin/muscle structure of human faces - Bonus points: Voice. So much of the sound of a person’s voice is about the shaped made by their mouth and throat (and torso). We could have an AI recreate the whole vocal path for a 3D character, resulting in more realistic voice acting.
- cyber_kinetist 5y agoI've seen musculoskeleton simulation (although passive and not active) used throughout VFX quite a lot, its basically what Weta Digital has been doing for years. (For example from the Hobbit movie 8 years ago: https://www.youtube.com/watch?v=r45e5Xky35k https://www.youtube.com/watch?v=r45e5Xky35k). Hell, Maya even has its own muscle system (Maya Muscle) that animators can already use out of the box. So the technology to simulate these kinds of things (at adequate realism for animations and movies) is already there. The problem is that it takes a painstakingly long time to do the modelling for these kinds of stuff. Anatomy is hard to get right, high quality commercial human template models cost you $10000+, and even then you're going to take a lot of time tweaking mesh geometry to make the simulation work. And the musculoskeletal models you're creating are all going to have different body shapes and might not even be human, so you need to create a lot of things from scratch (there was a recent paper on a musculoskeleton design tool for precisely this: https://www.dgp.toronto.edu/projects/modeling-musculoskeletal-anatomy https://www.dgp.toronto.edu/projects/modeling-musculoskeleta...) So you need a lot of work to do even just passive simulation (where the articulated skeleton just kinematically mimics the animation and the muscle effects are secondary) Though the bullet points you've listed are definitely all good research topics in itself. DeepMimic-style active musculoskeleton simulations (where models physically mimic given animation data using only muscle activations) are currently possible for line-segment muscles (the paper: https://mrl.snu.ac.kr/research/ProjectScalable/Page.htm https://mrl.snu.ac.kr/research/ProjectScalable/Page.htm) It uses Deep Reinforcement Learning with lots of training and fine-tuning though, it's not plug-and-play in its current state. Maybe such a similar thing can be also done with volumetric muscles too, but that's definitely going to be a hard paper to write since FEM simulation is incredibly costly and finicky to run stably, and you need a lot of simulation frames to use DRL. But I think the fundamental issue is: does this all have to do with actually making a good film/animation, outside of the realm of biomechanics? I really don't think so. The latest remastered Disney films all lack the fluidity of 2D animation of the past; the remade Lion King has one of the best realistically simulated animal characters but is utterly void of any artistic merit. And these graphical improvements don't seem to make the artist's life easier and is making animation even more labor-intensive. Nowadays it feels like artists in those big industries have less room to experiment because of all these "realistic" technical requirements. From this viewpoint, it makes sense why Disney/Pixar has still focused more on superficial skin deformation instead of a more realistic biomechanical approach: when making cartoonish 3D animations you need a lot of squash-and-stretch, and that is hard to do with rigid muscle systems. Better to model your character as a blob of mass, it's easier to animate, keyframe, and employ all kinds of effects to express artistic intent.
- anshumankmr 5y agoAt the 3:37 mark in the video on their channel, the last face for the older man looks creepy. https://youtu.be/k-RKSGbWLng?t=217 https://youtu.be/k-RKSGbWLng?t=217 He is the third person from the top. Are we sure that is the expression he made?
- ben_w 5y agoThat video has now been removed, but it looks like they replaced it with this: https://youtu.be/TwpLqTmvqVk https://youtu.be/TwpLqTmvqVk
- anshumankmr 5y agoThat was quick. I was just watching it when I wrote the comment.
- chillage 5y agoOriginal video link in the article doesn't seem to be working. This looks like a reupload of the same video, so far as I can guess https://www.youtube.com/watch?v=TwpLqTmvqVk https://www.youtube.com/watch?v=TwpLqTmvqVk
- smrtinsert 5y agoThanks! What still triggers it the uncanny aspect for me is that these faces are all way too symmetrical. Even the most attractive actors have more left-right variation. There's no doubt the light diffusion on the skin is looking way better in these though.
- deleted 5y ago[deleted]
- ridewinter 5y agoHow far away are we from rendering realistic humans in real-time?