23 ms·

Hyper-realistic digital humans in Unity

523 points by Naracion 5y ago

psyc 5y ago
More information about the production process and future plans here: https://80.lv/articles/enemies-the-working-process-behind-unity-s-new-tech-demo/ https://80.lv/articles/enemies-the-working-process-behind-un...
tobr 5y ago
That looks better to my eyes than Leia in Rogue One. Although it might help that I’m not familiar with this actress.
- gokhan 5y ago
  Check this for a fix for Leia in R1 https://www.youtube.com/watch?v=byKy9kGnyvo https://www.youtube.com/watch?v=byKy9kGnyvo
  - danparsonson 5y ago
    And his follow-up which is even better: https://www.youtube.com/watch?v=_CXMb_MO3aw https://www.youtube.com/watch?v=_CXMb_MO3aw
MasterScrat 5y ago
> Unity’s Demo productions drive advanced use of the Unity real-time 3D platform through autonomous creative projects, led by Creative Director Veselin Efremov. Does this mean this was rendered in real-time?
- swiftcoder 5y ago
  On a single RTX 3090, according to the announcement.
  - MasterScrat 5y ago
    Where did you see that? I would love a quote on this and CTRL-F returns nothing
    - yurymik 5y ago
      > With ultra settings, it runs at 4k 30fps (average of 40fps) on an i7 cpu and rtx 3090. by rob cupisz (Tech and Rendering Lead at Unity Demo Team) https://twitter.com/robcupisz/status/1505875759243612172 https://twitter.com/robcupisz/status/1505875759243612172
  - LeifCarrotson 5y ago
    I was going to say that the overlaps between one lock of hair to another, between hair and background/hair and skin, and the edges of lips and teeth looked a little poorly keyed, like the objects had high resolution textures but the surface map/motion rig was low-poly...which would be annoying but probably easy enough to ignore in an indie film. But live? On a single (hard to get) consumer GPU? That's seriously impressive. It makes me wonder how much of this is hand-tuned rigging and how much is physics based; if you tried to shake hands with this digital human using a game controller or VR rig, how would that look?
    - numlock86 5y ago
      Wait till you see Unreal's Matrix demo running live on a PS5 (even with interactive parts) instead of just some hand-crafted marketing material from Unity on YouTube yet again.
      - erikbye 5y ago
        Unity has released several assets from work shown on YouTube. E.g.:https://assetstore.unity.com/packages/essentials/tutorial-projects/the-heretic-digital-human-168620 https://assetstore.unity.com/packages/essentials/tutorial-pr...
        5y ago
- jerf 5y ago
  I remember for a long while everything was "Toy Story" quality in real time, the PS2, the PS3, etc. It never really was. But at some point, we definitely passed it. The room is nifty, but mostly been done. But that is a pretty good person. Lip sync is a bit off somehow... I think perhaps just too overexaggerated in the motions. But I couldn't tell you from a still frame that wasn't a real person, in real clothes. I also continue to find it amusing that we can build a person like that and sell it as commercial tech, but we still have to record people talking. (Though TTS has taken an interesting turn lately, after years of not much.)
  - zokier 5y ago
    > I also continue to find it amusing that we can build a person like that and sell it as commercial tech, but we still have to record people talking. (Though TTS has taken an interesting turn lately, after years of not much.) I suspect part of the problem is that dialog in games is still largely "static"; if the writing is pre-canned then it does not make that much sense to try to develop advanced tts to act it out. The situation will become interesting if we manage to produce sufficiently dynamic dialog system where it is not feasible to use pre-recorded voice acting anymore.
    - account42 5y ago
      Even with static writing, there is a huge cost to voice acting in dialogue-heavy games and good tts could help a lot here.
  - elliekelly 5y ago
    Tangentially related: Why can’t movie studios use a SnapChat-like filter to replace the mouth movement of foreign language films with the mouth movement of the voiceover actors? I feel like that technology definitely exists. There are so many great foreign movies & shows but the voiceover + unmatched mouth movement can be so distracting. Is it just too expensive?
    - Animats 5y ago
      That's available now. Can't find the link, but it exists at movie quality for non real time. It will probably become as routine as automatic dialog replacement.
PradeetPatel 5y ago
I wonder if this is a result of Unity putting their latest M&A - Weta Digital to good use.
xwdv 5y ago
Still shots look great but still looks too fake when in motion, we still have a long way to go.
- lostmsu 5y ago
  Yes, motion is highly uncorrelated. E.g. the whole body has no movement when she raises a hand with a chess piece. Humans don't do that. Everything would move from the chest up to assist the motion.
  - Raicuparta 5y ago
    It's motion captured, that's probably just how the actress moved. Real stuff looks fake all the time. The parts that look wrong in the animations are mostly due to small imprecisions, especially in the facial ones.
    - lostmsu 5y ago
      Do you have a source for motion capture?
    - xwdv 5y ago
      Why on earth would an actress move so fake then for something trying to be real? Yea let me dance like a robot for this hyper real CGI mo-cap.
seba_dos1 5y ago
The title seems editorialized by the submitter, and I don't think that's what the word "hyper" means.
- qwertox 5y ago
  It does have its meaning in this industry, just like in arts. https://en.wikipedia.org/wiki/Hyperrealism_(visual_arts) https://en.wikipedia.org/wiki/Hyperrealism_(visual_arts)
  - madacol 5y ago
    Sure, but Ctrl+F "hyper" = no result Title claims something the blog doesn't, and creates overexpectation
    - deleted 5y ago
      [deleted]
politelemon 5y ago
I can imagine at a certain point, with enough realism and detail and control, game engines becoming a viable content creation tool for pornography.
- jayd16 5y ago
  Oh my sweet summer child. Let's just say you aren't the first to think this.
  - jen729w 5y ago
    It occurred to me yesterday that at some point someone will inevitably advocate for the banning of deep fake pornography in their country by creating a porno starring the politicians of their lower house. I reckon it’d be illegal within the week.
    - throwawaygh 5y ago
      https://www.technologyreview.com/2021/02/12/1018222/deepfake-revenge-porn-coming-ban/ https://www.technologyreview.com/2021/02/12/1018222/deepfake...
      - johnnyanmac 5y ago
        Interesting, but I wonder how this would be enforced. I wouldn't be surprised if over half these cases were people who live in entirely different countries. would another country extradite a citizen over this? This has "technically" been possible for centuries so I wonder if there's any case study surrounding this.
    - azinman2 5y ago
      Such things already exist… they’re just not popular for obvious reasons.
      - dotancohen 5y ago
        What are the obvious reasons? I've heard "Fuck Trump" far more often than, say, "Fuck Kate Moss".
jonathan-adly 5y ago
A time-traveler from the 80's would be disappointed on how far we progressed technologically in everything, except video games. They would be amazed!
- 2III7 5y ago
  Electric cars/bicycles/scooters/motorcycles, computers in everyones pocket, huge cheap TV-s, fast wireless internet, robotic lawn mowers, radar cruise control and the list goes on and on and on...
  - TaylorAlexander 5y ago
    Drones that have a 30+ minute battery life. 4k cameras. Rockets that land themselves.
    - jpadkins 5y ago
      Are 4k cameras actually that much better than film from the 80s? (other than convenience)
      - adgjlsfhk1 5y ago
        depends what you mean by convenience. The image quality of a 4k camera isn't that much better, but that's largely because it doesn't need to be. Cameras in the 80s already produced great pictures. The difference is that they can weigh less than a pound and be run for hours at a time without worrying about paying ridiculous amounts for film.
      - dylan604 5y ago
        Hmmm, a 4K camera that can fly around for 30+ minutes with a controller attached to your cell phone (which in and of itself would blow the mind of any person in the 80s) that has the visual clarity of an image shot on 35mm motion picture camera weighing in at 70+lbs that could only be used aerially by renting a professionally piloted plane/helicopter at more $$ per hour rental than the drone you fly on your own costs? No, I can't imagine any benifits for modern tech from the 80s ;-)
freeCandy 5y ago
Link to the video: https://www.youtube.com/watch?v=eXYUNrgqWUU https://www.youtube.com/watch?v=eXYUNrgqWUU If you prefer not to accept cookies.
- jen729w 5y ago
  Just use a blocker everywhere if you prefer not to accept cookies? 1Blocker on Safari did the job for me here.
  - cgriswald 5y ago
    Your post is being downvoted because blocking cookies does not solve the problem. It is blocking cookies that causes the issue. When cookies are blocked (in my case, third-party cookies are blocked), an overlay appears over the video explaining the website that hosts the video will not allow the video to be played without a targeting cookie. FWIW, removing the overlay was not enough to get the video to play on the website; but I haven't looked any further into it—easier to just go to YouTube to play the video.
- isodev 5y ago
  Thanks! It's absurd how they make us pay in cookies to view their marketing PR posts.
- tinus_hn 5y ago
  How are you going to watch a YouTube video without accepting cookies?
  - jazzyjackson 5y ago
    Simple, YouTube sets the cookie without asking, no need to accept them ;)
    - tinus_hn 5y ago
      Perhaps it’s a EU thing but if I open the YouTube site I can’t proceed without accepting at least some cookies from the GDPR pop-up.
blueneko 5y ago
The hair looks really great! In the Matrix unreal engine demo, they kind of cheated by having a character with long hair.
- prometheus76 5y ago
  Funny sidenote about the Matrix unreal demo: where did Neo go? He just disappears.
enragedcacti 5y ago
This is really impressive assuming it is as automated/scalable as Unity implies. The skin and hair look great which are both really challenging for their own reasons, but one particular detail that was surprising was at 1:36. If you watch her lips closely you can see where the top and bottom lip stick together as they open. If they can simulate that level of skin detail when the motion is backed by mocap data this could be a huge quality jump for character-driven and dialogue heavy games. There are definitely still moments where the lips, teeth, and tongue look "3D" but I don't think I would notice if I weren't hyper focusing on the mouth and just enjoying the story.
- jcims 5y ago
  The lighting for teeth must be really hard to model. They alwys look wooden to me and still look kind of wooden here.
billconan 5y ago
is there a metahuman like generator? or we need to scan/model it?
- programd 5y ago
  https://www.unrealengine.com/en-US/digital-humans https://www.unrealengine.com/en-US/digital-humans
  - billconan 5y ago
    this is unreal engine. I was curious if unity has something similar?
Seattle3503 5y ago
This is cool from a developer perspective. I put my ganer hat on looked at their old demos. Demos always look great. I'll believe it when I see it in gameplay footage.
- johnnyanmac 5y ago
  These tech demos usually are for developers, so that makes sense. IME they tend to show off how tech will look 4-5 years down the line, about the time a modern AAA game needs to from production to release.
  - Seattle3503 5y ago
    That lines up with what I saw.
  - numlock86 5y ago
    > they tend to show off how tech will look 4-5 years down the line Unity's tech demo "Adam" is now more than 5 years old and I still don't see any Unity games coming even remotely close to that. Unity's tech demos are just hoaxes: Carefully crafted demos with custom engine parts and basically hand-written stuff like shaders and controllers all over. The irony is that these demos are primarily aimed at developers, yet you can't take a single piece of those things and just make it work on your own setup. In contrast when it comes to Unreal it's usually just a Ctrl+C and Ctrl+V away from success ...
    - johnnyanmac 5y ago
      Unity games? No, not to my knowledge. But the industry has certainly gotten extremely close to realizing the tech being leveraged in Adam. Some may even argue that some UE5 showcases with real samples have succeeded it, despite this being a non-real time showcase. I don't think this is because you can't just copy-paste an Adam asset into the Unity engine, however. AAA devs are going to leverage their own assets regardless. I think that's more of an image problem than a "technologically possible" problem, however. Many of Unity's huge successes come from the indie side, the Ori's, the Hollow Knights, the Cupheads. And it has a very big grasp on the mobile market. Even Pokemon Go decided to use Unity despite the IP being backed by a publisher known for their in-house engines. so a AAA studio isn't thinking of Unity for their next Call of Duty, but maybe for their Call of Duty mobile title. They seem to be trying to combat that sentiment with the likes of DOTS,HDRP, and whatever Ziva is trying to do, but those are still TBD.
monkeydust 5y ago
Is it strange but as a product guy my first thought when I saw this is how I could use it to replace my sales team... Well 80% of them at least.
- saas_sam 5y ago
  As a sales guy, you might want to learn a bit more about the value of sales. (It ain't the faces.)
naikrovek 5y ago
rendered hair gets better all the time, but it's still not there. sucks for me because I am weird and I find it hard not to pay close attention to hair in real life, sometimes. this makes flaws in rendered hair extremely obvious to me. :(
- the_af 5y ago
  Definitely, the hair effects are still not there. Something is off, the hair moves as a block in an unnatural way. The rest of the face is pretty good though. The eyes are weirdly intense, but I suppose it was a deliberate choice and some people with blue eyes look like that.
  - cgriswald 5y ago
    I agree about the hair. It looked great when it didn't move. I thought the eyes were great; to the extent that they were intense, she's an intense looking lady; and they may have done so to amplify the effects. I also thought the lips were great. It was the interplay between lips and teeth that really threw me off.
- Animats 5y ago
  rendered hair gets better all the time, but it's still not there. That's a sheer compute power problem. If you have a big enough render farm, you can run physics on every strand of hair.
  - Retr0id 5y ago
    Throwing more compute power at it is A solution, but it's not necessarily the only solution.
angryGhost 5y ago
And this was done in real time!? wow...
daenz 5y ago
The mouth movement[0] sold it for me. Mouths are incredibly hard to do in CG, because you have to have realistic interactions between lips, teeth, and tongue. Not sure how they scanned this so well (maybe it was hand animated?) but it looks excellent. 0. https://youtu.be/eXYUNrgqWUU?t=87 https://youtu.be/eXYUNrgqWUU?t=87
- otikik 5y ago
  Really? Because that was the worst part for me. To me it looks absolutely and positively synthetic. I mean, it is better than Terrance and Phillip from South Park, but perhaps because it's "close to reality, but not there yet" it falls into the uncanny valley to me. Perhaps this is similar to how some people aren't bothered by bad kerning (to which I'm fairly tolerant).
  - philovivero 5y ago
    You're right. The mouth movement was awful. I have no idea what OP is on about saying it was good. Maybe my uncanny valley is also steep, but nothing about this demo struck me as good. The tech maybe is good, but I wouldn't know it from this video. The animation and lighting are just awful.
    - ceeplusplus 5y ago
      Link in the OP states screen space global illumination was used, which is probably why the part where the character descends below the surface seems really fake. Still very behind Unreal Engine unfortunately, which already supports ray traced "real" global illumination.
polski-g 5y ago
I'm glad they're progressing on this. Would be really bad for the industry if Unreal was the only game in town.
- softfalcon 5y ago
  No kidding. I was a bit worried when Unreal was dropping demo after demo and Unity just remained silent for what felt like almost 3 years now.
mdoms 5y ago
Off topic but I really can't stand the increasingly common usage of the "hyper" modifier. What does it add to this deadline?
- slowhand09 5y ago
  Just added you to my list of hyperthinkers.
Semiapies 5y ago
"Look at our demo of a hyper-realistic human character!" eighteen million things happen around the character to distract the viewer's attention from the human character throughout most of the video Not the most confident tech demo. Looks like a reasonable degree of evolution, but as they start to get up toward turn-of-the-century movie CGI in games, now you run into the same issues that you see in mocap for movies--including that a lot of details of an actor's expressions and movements actually aren't captured, so you have to have animators go back and laboriously add all those lost details back in to have something that looks convincing. (Mind, I say turn-of-the-century, but man, the fundamental techniques for rendering skin convincingly have come a long way, even in current game engines.)
- ssully 5y ago
  Someone who knows better can correct me, but I assume they put the character in a complex environment to show that they are rendering a highly realistic character while still rendering a complex environment. I remember very impressive character rending tech demos from 10-20 years ago, but it was a single character in a static environment.
- MrLeap 5y ago
  They're showing off a lot of things at once here. The real time raytraced lighting on their vfx is impressive. To do that in tandem with their human while maintaining 30fps is pretty something. Every thing draws from the same well so to speak, a human that looked a little better but had to exist alone in a white room would have far less utility for real things.
- Animats 5y ago
  eighteen million things happen around the character to distract the viewer's attention from the human character throughout most of the video Yes. Why did they do all those crappy CG effects while showcasing photorealistic characters? Two people seated across a chessboard in a realistic room would have been more effective. It's getting to the point that everybody is doing this. Unreal Engine has Metahuman Creator.[1] Even Second Life has reasonably good heads now.[2] And has facial tracking on their roadmap. [2] https://www.unrealengine.com/en-US/metahuman-creator https://www.unrealengine.com/en-US/metahuman-creator [2] https://www.youtube.com/watch?v=OEjvD8C0g8w https://www.youtube.com/watch?v=OEjvD8C0g8w
hgomersall 5y ago
What's interesting is that for most of my childhood (80s and 90s), aside from a few wow moments I was pretty underwhelmed by the standard of the tech. It was like it was trying hard to be something it wasn't yet ready to be. I used to walk into television shops and think they all just look crap. Computers used to frustrate me so much - crunch crunch to do anything, and 256 colours was deemed good (!?). The first music players where you might be able to get one whole album onto a memory card that was too expensive to put a price in the rs catalogue (or perhaps too volatile a price). Anyway, tech was crap. Then around 2005 or something, it started becoming what it was meant to be. You could buy a computer and it could do everything you needed it to do; of course you always wanted more number crunching, but you could see where it was only just right the corner. Then GPUs started doing computation, and computation stopped being thought about. Memory was super fast and copious. One now felt limited by programming capability, not hardware. I'm now genuinely excited by technology. As much as it pains me to say it, the television departments are places of wonder. It's in that context that this post feels like another step towards achieving some promised vision. If this is realtime, it is truly fantastic. Now to hope we can deal with climate change and despots and poverty so it's not all for nought. As an aside, one early wow moment was the first time I saw a mini camcorder, then another for the first genuinely mobile phone; both Sony I think. Also, though a bit later, I remember seeing an in-car GPS and deciding it was basically a perfect interface for it's task.
- hnuser123456 5y ago
  If you're interested in tech becoming really magic, lookup the VRchat club scene. Real-life skilled DJs put out of clubs by covid, setup a twitch stream with a webcam on their DJ controller, while wearing full-body VR themselves, also streaming their DJ software and in-game view to an in-world stage screen in front of up to 80 guests, many if not most also wearing full-body VR, dancing along together, in any kind of digital world and wearing any kind of semi-humanoid (or not) 3D model you can imagine. The "metaverse" has already been running for a few years, but it's not Meta's, it's a thin multiplayer VR wrapper over Unity.
  - Cloudef 5y ago
    People alone in their rooms strapped screen into their face, just to be "together with other people" sounds more like dystopian nightmare to me
pistachiopro 5y ago
This is indeed rendered in realtime, but one thing to note is it's a "4D" capture, more-or-less meaning each frame of the animation is its own asset. This makes it possible to reproduce subtle physics like the lips sticking together slightly when the actor opens her mouth. The amount of storage space, alone, makes this impractical for anything other than demos. Unity claims they will be able to achieve this level of fidelity using a deep learning-based compression that will allow stuff like this to appear in game cutscenes, but all the movements will still be pre-baked. The only interaction possible will be moving the camera. At that point the technology will be very useful, but it's still a ways away from having such a realistic character that can react to you dynamically. (Though whether that's just a couple years of software technology progress, or a decade+ for hardware progress, who can say?)
- dylan604 5y ago
  >achieve this level of fidelity using a deep learning-based compression what does that mean? To me, they might have just as well said middle out compression.
  - pistachiopro 5y ago
    Unity recently acquired Ziva, which specializes in the detailed animation of humans and other animals. They were known for their (not realtime) physics-based solutions, but now they have an ML model for faces, apparently. As far as I know, it's still in beta and not widely available. Unity says they will re-release this demo with the Ziva face in a matter of weeks and the quality will be even higher. And possibly allowing interactivity as well?? I guess we'll see in a few weeks.
  - lagrange77 5y ago
    I guess this means dimensionality reduction for example with the use of a convolutional autoencoder.
  - greysphere 5y ago
    I'd guess a 4d version of: https://paperswithcode.com/method/nerf https://paperswithcode.com/method/nerf
davelondon 5y ago
Why is it that teeth always look weird? You'd think solid objects like that would be easy?
icu 5y ago
Are there any online educational resources that teach how to do this?
- wonderwonder 5y ago
  Just google Unity tutorial. If I remember correctly Unity uses C#. You are not going to be doing anything close to this out of the gate but I do recall it being a pretty easy environment to learn for novices.
owenfi 5y ago
Well...my days of thinking we're not quite out of the uncanny valley...are certainly coming to a middle. Granted, the gliding across an unwalkable moving floor gear contraption didn't help avoid the perception that I was looking at an automaton.
lagrange77 5y ago
Speeding up the eye blinking would decrease the uncannyness significantly.
hypertele-Xii 5y ago
As an artist and game developer it disappoints me how much resources are wasted in pursuit of "realism". Consider how art exploded when people realized they could paint something other than, you know, reality. Reality is boring. Get your realism off my games. Give me fantastical alien worlds to explore rendered in stylized and weird looks.
- softfalcon 5y ago
  Agree that I tend to prefer stylized non-photorealistic rendering in games. That being said, I think there’s room for both and luckily, the rendering learnings from one often improve the other. Nothing wrong with exploring both venues.
- whateveracct 5y ago
  You can't automate art. AAA is in large part about converting money into more money by way of video game sales. That's why it shouldn't be synonymous with anything elite.
  - Koshkin 5y ago
    Surely you can automate this: https://en.wikipedia.org/wiki/Black_Square_(painting) https://en.wikipedia.org/wiki/Black_Square_(painting)
    - whateveracct 5y ago
      you really couldn't
    - teawrecks 5y ago
      You could always say "I could have made that", but the important part is: you didn't. No one cares if you automate the painting of a black square. They care about the intention of the mind that thought it was important enough to do. That's why it's art.
    - creata 5y ago
      Good luck automating the part where you convince people it's worth something.
LAC-Tech 5y ago
Really impressive. I still don't think it's out of the uncanny valley, but it's definitely climbing out the other side.
FpUser 5y ago
There are things that look realistic from a first glance, like hires skin and hair. But face animation - no way. It is freakish and screams fake.
otikik 5y ago
They are Hyper-Realistic Digital Stills of Humans. There's a lot of room for improvement in the animation department, though. For me the worst offender is the mouth, it moves in a completely unrealistic manner. The second worst offender is the head/neck movement, which moves with robot-like precision. Finally, the eyes, which are (granted) not as "dead" as in other models, stare too much and communicate too little.
- kingcharles 5y ago
  The hand movements were jerky too. Just like they were in the 1990s. I don't understand why we're still struggling with producing animation that isn't uncanny valley?
  - subb 5y ago
    Most hand animations are done with keyframes, because mocapping hands is difficult. And with keyframes animation, much like modelling geometry traditionally vs photogrammetry, we can't produce the needed amount of details in animations for them to look real. Moreover, we are extremely good at spotting something off with hands movement.
- drunkenmagician 5y ago
  Mouth movements were not 100% either. FWIW mouth animation is actually quite hard to get right, I remember artists at a previous company I worked at (also doing super realistic digital humans) spending hours try to get it just right, but still always seem to be able notice it being a bit off...
torginus 5y ago
I'm kinda confused why Unity keeps doing this - they keep putting out high-end demo after high-end demo, but that's not where there core userbase is. Their main users are people who build games for phones and indies, with basically zero usage in the AAA space. And Unity's performance/stability is still not that great afaik. It seems to me that they are trying to prove that they are a serious 'AAA' engine, but these demos aren't that convincing to me - AAA is a lot more than putting fancy shaders on high-poly models, it's about handling huge amounts of objects in a dynamics situation, displaying large worlds via streaming, having a workflow that accommodates every creative professional, and offering great performance and visuals even on very complex scenes. I've hear that even these highly impressive demos are fake - they built a ton of custom code for each one they rebuild core Unity features, meaning if you wanted to replicate this for yourself, you'd be in for a ton of development. Comparatively UE5's Nanite demo showed off a tech that's ready to go for production.
- belval 5y ago
  Pretty sure they want to break out of the core userbase. I once interviewed there and they really stressed that they were a 3D platform and not just a game engine. In that context it would make sense to make demos for the capabilities that people don't know that you have.
- mrtweetyhack 5y ago
- Etheryte 5y ago
  I don't work in the industry, but as an outsider your question seems to answer itself. I would wager that Unity wants to take the step from the hobbyist space to the professional space — because surely that's where the bigger sums are made. Even if they're not there right now, consistently putting out content that indicates that you're working on it is a great way to captivate your audience over time and shift the perception of what your brand is about. In the startup space, this would be similar to building in the open — it helps you signal your brand, build a reputation and, hopefully, build a customer base that matches your sell.
- 5y ago
gigatexal 5y ago
Nope. Nope. Nope. Do not want. That’s too good. It’s creepy good. I need to know what’s real and what isn’t. This is so real that I’m freaking out a bit.
- bhedgeoser 5y ago
  She wasn't breathing.
playpause 5y ago
How come the latest cutting edge tech demos showcasing photorealism are barely more ‘realistic’ than a PS4 video game cutscene from a few years ago? There wasn’t any part of the demo video where the guy looked or moved like a human.
- CanSpice 5y ago
  Those cutscenes weren't rendered in real-time, instead taking longer than one second to render one second of video. This one is rendered in real-time. It's impressive for sure to render lifelike humans, but to do so in real-time is even more impressive.
  - jay_kyburz 5y ago
    Yes, but that's the point. The should have released it as an App that we can interact with, pan around, move the chess pieces. It might as well just be a video. Actually they should have plugged in stockfish and let us play against their character.
  - whywhywhywhy 5y ago
    > Those cutscenes weren't rendered in real-time Don't most games use realtime cutscenes these days? Nothing shown here is a gigantic leap to what you see in Death Stranding for example. I'm not looking at her and seeing an actress it still just looks like a very good 3D model.
davidhyde 5y ago
It does not look like they simulate the muscles underneath the skin which makes it feel uncanny unfortunately. Close though! Very close and impressive indeed. Static rendering is excellent.
farzher 5y ago
newobj 5y ago
Nope, still the uncanny valley
cercatrova 5y ago
Interesting comment from a game developer on /r/pcgaming [0] > The problem with Unity cinematics is that they are much less representative of their tools and engine out-of-the-box capabilities than Unreal's demos. > They use heavily customized assets specifically for a video and even do a lot of custom changes to the actual engine held by hot glue and duct tape to make it work. Then you try to recreate only fragments of what was show and nothing works. Even their released demos are usually not very portable and useful for devs, while in Unreal you can pretty much copy useful modules and most of it just works. This is why devs struggle to make things that look like demos Unity released 5 years ago... > Unreal is less smoke and mirror. They focus more on things that are actually available inside the editor (eg. the main character in Matrix Awakens was taken straight from the metahuman editor instead of spending hundreds of thousands of dollars on custom scanned model rigged by a huge team of world experts). > Unity does much better job with their demos to impress general public, but look at comments on their own forums and you will see over and over again how many devs are frustrated with their marketing tricks that don't turn into something real. > Some devs say that this difference comes from the fact that Unreal guys have to eat their own food that they make, because they also make games (like Fortnite), while Unity is more detached from actual needs of game devs and recently they became obsessed with film industry, even acquired Weta Digital that made LOTR and Avatar. It should be exciting and impressive, but the comments from their licensees (waiting for years to fix core issues) were quite negative. [0] http://www.reddit.com/r/pcgaming/comments/tjasoe/enemies_realtime_tech_demo_by_unity/i1jcyjr http://www.reddit.com/r/pcgaming/comments/tjasoe/enemies_rea...
- deleted 5y ago
  [deleted]
- squeegmeister 5y ago
  The "glued together" may be a little overstated. Their "Mega-City" demoed in 2019 is available to download and run as a standard unity project. I haven't looked at any of the code, but the fact that it runs on a mainline version of unity and not some forked, hacked together version should count for something.
ourmandave 5y ago
Demo released just in time for the Game Developers Conference, which they'll be doing a session at on March 24th. https://schedule.gdconf.com/session/inside-unitys-new-flagship-demo-presented-by-unity/887041 https://schedule.gdconf.com/session/inside-unitys-new-flagsh...
slackstation 5y ago
The eyes still look dead. All the raytracing in the world doesn't matter if the eyes don't look like they have life.
throwoutway 5y ago
I want to make a demo like this but I don’t even know where to start. Does it take thousands of hours of skills and training to become this good at demos?
jacquesm 5y ago
The illusion was pretty good until she started talking and then it fell apart completely for me.
ehsankia 5y ago
Am I the only one who finds that lipsync is ALWAYS off in every single hyper-realistic game / demo? Is lipsync also subject to uncanny valley or what? Why is it so damn hard to sync audio with lips in these simulations? Go to 1:34 in the video, slow it down to 0.25x on Youtube, and press play.
- erikbye 5y ago
  Lip syncing in games continues to disappoint. LA Noire from 2011 had some artifacts, but it's still better than most games since.
patientplatypus 5y ago
mr-pink 5y ago
what's the point of this? they had hyper realistic humans back in Myst back in 1993
yahn00 5y ago
I can find hyper-realistic humans anywhere, why waste computing power on this?
batch12 5y ago
Creepy. Most of these human renderings look dead to me.
stevev 5y ago
That’s great. Other studios are working on realistic problems and performance as well like Pearl Abyss. Difference is, they actually have games using the tech.
tw20212021 5y ago
What a stupid cookie popup. So a cookie from 'livecasinocomparer.com' is a Strictly Necessary Cookie. WTH
sharkster711 5y ago
Isn't that Carrie-Anne Moss?
alkonaut 5y ago
In-engine is not in-game. I'm not that interested in what a game engine can do unless it's also constrained by doing all the other things a game engine can do. Show me what's possible in unity at 100fps on a consumer system while doing normal game loop things as well.
shetill 5y ago
yet another project that will stay in preview forever
deleted 5y ago
[deleted]
aniijbod 5y ago
Most of the comments that I can see below are restricted to discussing the impact of announcement and demo on the Unity vs. Unreal situation. However, it turns out that here are other very relevant and important issues which arise from the technological specifics behind the demo which, although they obviously directly impinge on the rivalry, have potentially broader implications. 'Uncanny valley' issues related to face-based (as well as body motion-based) realism in simulation for both games development and video production, whilst there have been recent very significant advances in addressing them are still a (pun unavoidable) rapidly moving target, a target where this demo highlights both substantial advances and still as-yet unresolved serious shortcomings. The advances mostly relate to the introduction of machine learning and 15TB of training data into the content production workflow to reduce a rendering job from 6 hours per 50 frames to 3 milliseconds per frame as detailed in this article: https://www.globenewswire.com/news-release/2021/12/10/2349777/0/en/Senua-s-Saga-Hellblade-II-Gameplay-Reveals-Character-by-Ziva-Dynamics.html https://www.globenewswire.com/news-release/2021/12/10/234977... Returning for just a moment to the UE vs Unreal thing, those results were discussed in an article about a demo created using Ziva Dynamics's Real Time Trainer and Unreal Engine 4.26, but Ziva was subsequently acquired by Unity. The discussion I'd be interested to see here (or somewhere else!) is about how ZRT compares with (or perhaps even fits into?) Unreal Engine's Metahuman workflow and resulting content creation and realism results. Perhaps more importantly, how might the ML-driven capabilities of ZRT (will they still continue to be available under Unreal as well as Unity?) and ML-based impact upon realism be expected to manifest themselves Here's a link to the ZRT 'preparation workflow', which a Z-brush user who was trialling the trainer setup says took them about an hour to produce a workable result: https://drive.google.com/uc?export=download&id=15J6vX-6vU5EUuUDhvfrfDb-ZtoDfwcqg https://drive.google.com/uc?export=download&id=15J6vX-6vU5EU...
chakintosh 5y ago
Not as uncanny valley as UE's Metahumans.
globular-toast 5y ago
I've noticed that the more "realistic" animated humans become, the less conventionally attractive they become. I began to notice this years ago when characters started to have skin blemishes etc. It struck me as a simple gimmick then. Like all they could do was increase the texture resolution and to show that off you need "interesting" facial features. All of this is just trickery to distract you from the uncanny valley. The real test of this rendering is "perfect" faces like Charlize Theron or George Clooney. Those are real people too, by the way. I also notice that the latest "hyper-realistic" models are almost always female too. I wonder why that is.
sinuhe69 5y ago
From afar, the eyes seem to bulge out and up-close the hairs and hairs movement are still quite artificial, though.

523 points by Naracion 5y ago

psyc 5y ago

More information about the production process and future plans here: https://80.lv/articles/enemies-the-working-process-behind-unity-s-new-tech-demo/ https://80.lv/articles/enemies-the-working-process-behind-un...

tobr 5y ago

That looks better to my eyes than Leia in Rogue One. Although it might help that I’m not familiar with this actress.

gokhan 5y ago
Check this for a fix for Leia in R1 https://www.youtube.com/watch?v=byKy9kGnyvo https://www.youtube.com/watch?v=byKy9kGnyvo
- danparsonson 5y ago
  And his follow-up which is even better: https://www.youtube.com/watch?v=_CXMb_MO3aw https://www.youtube.com/watch?v=_CXMb_MO3aw

MasterScrat 5y ago

> Unity’s Demo productions drive advanced use of the Unity real-time 3D platform through autonomous creative projects, led by Creative Director Veselin Efremov. Does this mean this was rendered in real-time?

swiftcoder 5y ago
On a single RTX 3090, according to the announcement.
- MasterScrat 5y ago
  Where did you see that? I would love a quote on this and CTRL-F returns nothing
  - yurymik 5y ago
    > With ultra settings, it runs at 4k 30fps (average of 40fps) on an i7 cpu and rtx 3090. by rob cupisz (Tech and Rendering Lead at Unity Demo Team) https://twitter.com/robcupisz/status/1505875759243612172 https://twitter.com/robcupisz/status/1505875759243612172
- LeifCarrotson 5y ago
  I was going to say that the overlaps between one lock of hair to another, between hair and background/hair and skin, and the edges of lips and teeth looked a little poorly keyed, like the objects had high resolution textures but the surface map/motion rig was low-poly...which would be annoying but probably easy enough to ignore in an indie film. But live? On a single (hard to get) consumer GPU? That's seriously impressive. It makes me wonder how much of this is hand-tuned rigging and how much is physics based; if you tried to shake hands with this digital human using a game controller or VR rig, how would that look?
  - numlock86 5y ago
    Wait till you see Unreal's Matrix demo running live on a PS5 (even with interactive parts) instead of just some hand-crafted marketing material from Unity on YouTube yet again.
    - erikbye 5y ago
      Unity has released several assets from work shown on YouTube. E.g.:https://assetstore.unity.com/packages/essentials/tutorial-projects/the-heretic-digital-human-168620 https://assetstore.unity.com/packages/essentials/tutorial-pr...
      - 5y ago
jerf 5y ago
I remember for a long while everything was "Toy Story" quality in real time, the PS2, the PS3, etc. It never really was. But at some point, we definitely passed it. The room is nifty, but mostly been done. But that is a pretty good person. Lip sync is a bit off somehow... I think perhaps just too overexaggerated in the motions. But I couldn't tell you from a still frame that wasn't a real person, in real clothes. I also continue to find it amusing that we can build a person like that and sell it as commercial tech, but we still have to record people talking. (Though TTS has taken an interesting turn lately, after years of not much.)
- zokier 5y ago
  > I also continue to find it amusing that we can build a person like that and sell it as commercial tech, but we still have to record people talking. (Though TTS has taken an interesting turn lately, after years of not much.) I suspect part of the problem is that dialog in games is still largely "static"; if the writing is pre-canned then it does not make that much sense to try to develop advanced tts to act it out. The situation will become interesting if we manage to produce sufficiently dynamic dialog system where it is not feasible to use pre-recorded voice acting anymore.
  - account42 5y ago
    Even with static writing, there is a huge cost to voice acting in dialogue-heavy games and good tts could help a lot here.
- elliekelly 5y ago
  Tangentially related: Why can’t movie studios use a SnapChat-like filter to replace the mouth movement of foreign language films with the mouth movement of the voiceover actors? I feel like that technology definitely exists. There are so many great foreign movies & shows but the voiceover + unmatched mouth movement can be so distracting. Is it just too expensive?
  - Animats 5y ago
    That's available now. Can't find the link, but it exists at movie quality for non real time. It will probably become as routine as automatic dialog replacement.

PradeetPatel 5y ago

I wonder if this is a result of Unity putting their latest M&A - Weta Digital to good use.

xwdv 5y ago

Still shots look great but still looks too fake when in motion, we still have a long way to go.

lostmsu 5y ago
Yes, motion is highly uncorrelated. E.g. the whole body has no movement when she raises a hand with a chess piece. Humans don't do that. Everything would move from the chest up to assist the motion.
- Raicuparta 5y ago
  It's motion captured, that's probably just how the actress moved. Real stuff looks fake all the time. The parts that look wrong in the animations are mostly due to small imprecisions, especially in the facial ones.
  - lostmsu 5y ago
    Do you have a source for motion capture?
  - xwdv 5y ago
    Why on earth would an actress move so fake then for something trying to be real? Yea let me dance like a robot for this hyper real CGI mo-cap.

seba_dos1 5y ago

The title seems editorialized by the submitter, and I don't think that's what the word "hyper" means.

qwertox 5y ago
It does have its meaning in this industry, just like in arts. https://en.wikipedia.org/wiki/Hyperrealism_(visual_arts) https://en.wikipedia.org/wiki/Hyperrealism_(visual_arts)
- madacol 5y ago
  Sure, but Ctrl+F "hyper" = no result Title claims something the blog doesn't, and creates overexpectation
  - deleted 5y ago
    [deleted]

politelemon 5y ago

I can imagine at a certain point, with enough realism and detail and control, game engines becoming a viable content creation tool for pornography.

jayd16 5y ago
Oh my sweet summer child. Let's just say you aren't the first to think this.
- jen729w 5y ago
  It occurred to me yesterday that at some point someone will inevitably advocate for the banning of deep fake pornography in their country by creating a porno starring the politicians of their lower house. I reckon it’d be illegal within the week.
  - throwawaygh 5y ago
    https://www.technologyreview.com/2021/02/12/1018222/deepfake-revenge-porn-coming-ban/ https://www.technologyreview.com/2021/02/12/1018222/deepfake...
    - johnnyanmac 5y ago
      Interesting, but I wonder how this would be enforced. I wouldn't be surprised if over half these cases were people who live in entirely different countries. would another country extradite a citizen over this? This has "technically" been possible for centuries so I wonder if there's any case study surrounding this.
  - azinman2 5y ago
    Such things already exist… they’re just not popular for obvious reasons.
    - dotancohen 5y ago
      What are the obvious reasons? I've heard "Fuck Trump" far more often than, say, "Fuck Kate Moss".

jonathan-adly 5y ago

A time-traveler from the 80's would be disappointed on how far we progressed technologically in everything, except video games. They would be amazed!

2III7 5y ago
Electric cars/bicycles/scooters/motorcycles, computers in everyones pocket, huge cheap TV-s, fast wireless internet, robotic lawn mowers, radar cruise control and the list goes on and on and on...
- TaylorAlexander 5y ago
  Drones that have a 30+ minute battery life. 4k cameras. Rockets that land themselves.
  - jpadkins 5y ago
    Are 4k cameras actually that much better than film from the 80s? (other than convenience)
    - adgjlsfhk1 5y ago
      depends what you mean by convenience. The image quality of a 4k camera isn't that much better, but that's largely because it doesn't need to be. Cameras in the 80s already produced great pictures. The difference is that they can weigh less than a pound and be run for hours at a time without worrying about paying ridiculous amounts for film.
    - dylan604 5y ago
      Hmmm, a 4K camera that can fly around for 30+ minutes with a controller attached to your cell phone (which in and of itself would blow the mind of any person in the 80s) that has the visual clarity of an image shot on 35mm motion picture camera weighing in at 70+lbs that could only be used aerially by renting a professionally piloted plane/helicopter at more $$ per hour rental than the drone you fly on your own costs? No, I can't imagine any benifits for modern tech from the 80s ;-)

freeCandy 5y ago

Link to the video: https://www.youtube.com/watch?v=eXYUNrgqWUU https://www.youtube.com/watch?v=eXYUNrgqWUU If you prefer not to accept cookies.

jen729w 5y ago
Just use a blocker everywhere if you prefer not to accept cookies? 1Blocker on Safari did the job for me here.
- cgriswald 5y ago
  Your post is being downvoted because blocking cookies does not solve the problem. It is blocking cookies that causes the issue. When cookies are blocked (in my case, third-party cookies are blocked), an overlay appears over the video explaining the website that hosts the video will not allow the video to be played without a targeting cookie. FWIW, removing the overlay was not enough to get the video to play on the website; but I haven't looked any further into it—easier to just go to YouTube to play the video.
isodev 5y ago
Thanks! It's absurd how they make us pay in cookies to view their marketing PR posts.
tinus_hn 5y ago
How are you going to watch a YouTube video without accepting cookies?
- jazzyjackson 5y ago
  Simple, YouTube sets the cookie without asking, no need to accept them ;)
  - tinus_hn 5y ago
    Perhaps it’s a EU thing but if I open the YouTube site I can’t proceed without accepting at least some cookies from the GDPR pop-up.

blueneko 5y ago

The hair looks really great! In the Matrix unreal engine demo, they kind of cheated by having a character with long hair.

prometheus76 5y ago
Funny sidenote about the Matrix unreal demo: where did Neo go? He just disappears.

enragedcacti 5y ago

This is really impressive assuming it is as automated/scalable as Unity implies. The skin and hair look great which are both really challenging for their own reasons, but one particular detail that was surprising was at 1:36. If you watch her lips closely you can see where the top and bottom lip stick together as they open. If they can simulate that level of skin detail when the motion is backed by mocap data this could be a huge quality jump for character-driven and dialogue heavy games. There are definitely still moments where the lips, teeth, and tongue look "3D" but I don't think I would notice if I weren't hyper focusing on the mouth and just enjoying the story.

jcims 5y ago
The lighting for teeth must be really hard to model. They alwys look wooden to me and still look kind of wooden here.

billconan 5y ago

is there a metahuman like generator? or we need to scan/model it?

programd 5y ago
https://www.unrealengine.com/en-US/digital-humans https://www.unrealengine.com/en-US/digital-humans
- billconan 5y ago
  this is unreal engine. I was curious if unity has something similar?

Seattle3503 5y ago

This is cool from a developer perspective. I put my ganer hat on looked at their old demos. Demos always look great. I'll believe it when I see it in gameplay footage.

johnnyanmac 5y ago
These tech demos usually are for developers, so that makes sense. IME they tend to show off how tech will look 4-5 years down the line, about the time a modern AAA game needs to from production to release.
- Seattle3503 5y ago
  That lines up with what I saw.
- numlock86 5y ago
  > they tend to show off how tech will look 4-5 years down the line Unity's tech demo "Adam" is now more than 5 years old and I still don't see any Unity games coming even remotely close to that. Unity's tech demos are just hoaxes: Carefully crafted demos with custom engine parts and basically hand-written stuff like shaders and controllers all over. The irony is that these demos are primarily aimed at developers, yet you can't take a single piece of those things and just make it work on your own setup. In contrast when it comes to Unreal it's usually just a Ctrl+C and Ctrl+V away from success ...
  - johnnyanmac 5y ago
    Unity games? No, not to my knowledge. But the industry has certainly gotten extremely close to realizing the tech being leveraged in Adam. Some may even argue that some UE5 showcases with real samples have succeeded it, despite this being a non-real time showcase. I don't think this is because you can't just copy-paste an Adam asset into the Unity engine, however. AAA devs are going to leverage their own assets regardless. I think that's more of an image problem than a "technologically possible" problem, however. Many of Unity's huge successes come from the indie side, the Ori's, the Hollow Knights, the Cupheads. And it has a very big grasp on the mobile market. Even Pokemon Go decided to use Unity despite the IP being backed by a publisher known for their in-house engines. so a AAA studio isn't thinking of Unity for their next Call of Duty, but maybe for their Call of Duty mobile title. They seem to be trying to combat that sentiment with the likes of DOTS,HDRP, and whatever Ziva is trying to do, but those are still TBD.

monkeydust 5y ago

Is it strange but as a product guy my first thought when I saw this is how I could use it to replace my sales team... Well 80% of them at least.

saas_sam 5y ago
As a sales guy, you might want to learn a bit more about the value of sales. (It ain't the faces.)

naikrovek 5y ago

rendered hair gets better all the time, but it's still not there. sucks for me because I am weird and I find it hard not to pay close attention to hair in real life, sometimes. this makes flaws in rendered hair extremely obvious to me. :(

the_af 5y ago
Definitely, the hair effects are still not there. Something is off, the hair moves as a block in an unnatural way. The rest of the face is pretty good though. The eyes are weirdly intense, but I suppose it was a deliberate choice and some people with blue eyes look like that.
- cgriswald 5y ago
  I agree about the hair. It looked great when it didn't move. I thought the eyes were great; to the extent that they were intense, she's an intense looking lady; and they may have done so to amplify the effects. I also thought the lips were great. It was the interplay between lips and teeth that really threw me off.
Animats 5y ago
rendered hair gets better all the time, but it's still not there. That's a sheer compute power problem. If you have a big enough render farm, you can run physics on every strand of hair.
- Retr0id 5y ago
  Throwing more compute power at it is A solution, but it's not necessarily the only solution.

angryGhost 5y ago

And this was done in real time!? wow...

daenz 5y ago

The mouth movement[0] sold it for me. Mouths are incredibly hard to do in CG, because you have to have realistic interactions between lips, teeth, and tongue. Not sure how they scanned this so well (maybe it was hand animated?) but it looks excellent. 0. https://youtu.be/eXYUNrgqWUU?t=87 https://youtu.be/eXYUNrgqWUU?t=87

otikik 5y ago
Really? Because that was the worst part for me. To me it looks absolutely and positively synthetic. I mean, it is better than Terrance and Phillip from South Park, but perhaps because it's "close to reality, but not there yet" it falls into the uncanny valley to me. Perhaps this is similar to how some people aren't bothered by bad kerning (to which I'm fairly tolerant).
- philovivero 5y ago
  You're right. The mouth movement was awful. I have no idea what OP is on about saying it was good. Maybe my uncanny valley is also steep, but nothing about this demo struck me as good. The tech maybe is good, but I wouldn't know it from this video. The animation and lighting are just awful.
  - ceeplusplus 5y ago
    Link in the OP states screen space global illumination was used, which is probably why the part where the character descends below the surface seems really fake. Still very behind Unreal Engine unfortunately, which already supports ray traced "real" global illumination.

polski-g 5y ago

I'm glad they're progressing on this. Would be really bad for the industry if Unreal was the only game in town.

softfalcon 5y ago
No kidding. I was a bit worried when Unreal was dropping demo after demo and Unity just remained silent for what felt like almost 3 years now.

mdoms 5y ago

Off topic but I really can't stand the increasingly common usage of the "hyper" modifier. What does it add to this deadline?

slowhand09 5y ago
Just added you to my list of hyperthinkers.

Semiapies 5y ago

"Look at our demo of a hyper-realistic human character!" eighteen million things happen around the character to distract the viewer's attention from the human character throughout most of the video Not the most confident tech demo. Looks like a reasonable degree of evolution, but as they start to get up toward turn-of-the-century movie CGI in games, now you run into the same issues that you see in mocap for movies--including that a lot of details of an actor's expressions and movements actually aren't captured, so you have to have animators go back and laboriously add all those lost details back in to have something that looks convincing. (Mind, I say turn-of-the-century, but man, the fundamental techniques for rendering skin convincingly have come a long way, even in current game engines.)

ssully 5y ago
Someone who knows better can correct me, but I assume they put the character in a complex environment to show that they are rendering a highly realistic character while still rendering a complex environment. I remember very impressive character rending tech demos from 10-20 years ago, but it was a single character in a static environment.
MrLeap 5y ago
They're showing off a lot of things at once here. The real time raytraced lighting on their vfx is impressive. To do that in tandem with their human while maintaining 30fps is pretty something. Every thing draws from the same well so to speak, a human that looked a little better but had to exist alone in a white room would have far less utility for real things.
Animats 5y ago
eighteen million things happen around the character to distract the viewer's attention from the human character throughout most of the video Yes. Why did they do all those crappy CG effects while showcasing photorealistic characters? Two people seated across a chessboard in a realistic room would have been more effective. It's getting to the point that everybody is doing this. Unreal Engine has Metahuman Creator.[1] Even Second Life has reasonably good heads now.[2] And has facial tracking on their roadmap. [2] https://www.unrealengine.com/en-US/metahuman-creator https://www.unrealengine.com/en-US/metahuman-creator [2] https://www.youtube.com/watch?v=OEjvD8C0g8w https://www.youtube.com/watch?v=OEjvD8C0g8w

hgomersall 5y ago

What's interesting is that for most of my childhood (80s and 90s), aside from a few wow moments I was pretty underwhelmed by the standard of the tech. It was like it was trying hard to be something it wasn't yet ready to be. I used to walk into television shops and think they all just look crap. Computers used to frustrate me so much - crunch crunch to do anything, and 256 colours was deemed good (!?). The first music players where you might be able to get one whole album onto a memory card that was too expensive to put a price in the rs catalogue (or perhaps too volatile a price). Anyway, tech was crap. Then around 2005 or something, it started becoming what it was meant to be. You could buy a computer and it could do everything you needed it to do; of course you always wanted more number crunching, but you could see where it was only just right the corner. Then GPUs started doing computation, and computation stopped being thought about. Memory was super fast and copious. One now felt limited by programming capability, not hardware. I'm now genuinely excited by technology. As much as it pains me to say it, the television departments are places of wonder. It's in that context that this post feels like another step towards achieving some promised vision. If this is realtime, it is truly fantastic. Now to hope we can deal with climate change and despots and poverty so it's not all for nought. As an aside, one early wow moment was the first time I saw a mini camcorder, then another for the first genuinely mobile phone; both Sony I think. Also, though a bit later, I remember seeing an in-car GPS and deciding it was basically a perfect interface for it's task.

hnuser123456 5y ago
If you're interested in tech becoming really magic, lookup the VRchat club scene. Real-life skilled DJs put out of clubs by covid, setup a twitch stream with a webcam on their DJ controller, while wearing full-body VR themselves, also streaming their DJ software and in-game view to an in-world stage screen in front of up to 80 guests, many if not most also wearing full-body VR, dancing along together, in any kind of digital world and wearing any kind of semi-humanoid (or not) 3D model you can imagine. The "metaverse" has already been running for a few years, but it's not Meta's, it's a thin multiplayer VR wrapper over Unity.
- Cloudef 5y ago
  People alone in their rooms strapped screen into their face, just to be "together with other people" sounds more like dystopian nightmare to me

pistachiopro 5y ago

This is indeed rendered in realtime, but one thing to note is it's a "4D" capture, more-or-less meaning each frame of the animation is its own asset. This makes it possible to reproduce subtle physics like the lips sticking together slightly when the actor opens her mouth. The amount of storage space, alone, makes this impractical for anything other than demos. Unity claims they will be able to achieve this level of fidelity using a deep learning-based compression that will allow stuff like this to appear in game cutscenes, but all the movements will still be pre-baked. The only interaction possible will be moving the camera. At that point the technology will be very useful, but it's still a ways away from having such a realistic character that can react to you dynamically. (Though whether that's just a couple years of software technology progress, or a decade+ for hardware progress, who can say?)

dylan604 5y ago
>achieve this level of fidelity using a deep learning-based compression what does that mean? To me, they might have just as well said middle out compression.
- pistachiopro 5y ago
  Unity recently acquired Ziva, which specializes in the detailed animation of humans and other animals. They were known for their (not realtime) physics-based solutions, but now they have an ML model for faces, apparently. As far as I know, it's still in beta and not widely available. Unity says they will re-release this demo with the Ziva face in a matter of weeks and the quality will be even higher. And possibly allowing interactivity as well?? I guess we'll see in a few weeks.
- lagrange77 5y ago
  I guess this means dimensionality reduction for example with the use of a convolutional autoencoder.
- greysphere 5y ago
  I'd guess a 4d version of: https://paperswithcode.com/method/nerf https://paperswithcode.com/method/nerf

davelondon 5y ago

Why is it that teeth always look weird? You'd think solid objects like that would be easy?

icu 5y ago

Are there any online educational resources that teach how to do this?

wonderwonder 5y ago
Just google Unity tutorial. If I remember correctly Unity uses C#. You are not going to be doing anything close to this out of the gate but I do recall it being a pretty easy environment to learn for novices.

owenfi 5y ago

Well...my days of thinking we're not quite out of the uncanny valley...are certainly coming to a middle. Granted, the gliding across an unwalkable moving floor gear contraption didn't help avoid the perception that I was looking at an automaton.

lagrange77 5y ago

Speeding up the eye blinking would decrease the uncannyness significantly.

hypertele-Xii 5y ago

As an artist and game developer it disappoints me how much resources are wasted in pursuit of "realism". Consider how art exploded when people realized they could paint something other than, you know, reality. Reality is boring. Get your realism off my games. Give me fantastical alien worlds to explore rendered in stylized and weird looks.

softfalcon 5y ago
Agree that I tend to prefer stylized non-photorealistic rendering in games. That being said, I think there’s room for both and luckily, the rendering learnings from one often improve the other. Nothing wrong with exploring both venues.
whateveracct 5y ago
You can't automate art. AAA is in large part about converting money into more money by way of video game sales. That's why it shouldn't be synonymous with anything elite.
- Koshkin 5y ago
  Surely you can automate this: https://en.wikipedia.org/wiki/Black_Square_(painting) https://en.wikipedia.org/wiki/Black_Square_(painting)
  - whateveracct 5y ago
    you really couldn't
  - teawrecks 5y ago
    You could always say "I could have made that", but the important part is: you didn't. No one cares if you automate the painting of a black square. They care about the intention of the mind that thought it was important enough to do. That's why it's art.
  - creata 5y ago
    Good luck automating the part where you convince people it's worth something.

LAC-Tech 5y ago

Really impressive. I still don't think it's out of the uncanny valley, but it's definitely climbing out the other side.

FpUser 5y ago

There are things that look realistic from a first glance, like hires skin and hair. But face animation - no way. It is freakish and screams fake.

otikik 5y ago

They are Hyper-Realistic Digital Stills of Humans. There's a lot of room for improvement in the animation department, though. For me the worst offender is the mouth, it moves in a completely unrealistic manner. The second worst offender is the head/neck movement, which moves with robot-like precision. Finally, the eyes, which are (granted) not as "dead" as in other models, stare too much and communicate too little.

kingcharles 5y ago
The hand movements were jerky too. Just like they were in the 1990s. I don't understand why we're still struggling with producing animation that isn't uncanny valley?
- subb 5y ago
  Most hand animations are done with keyframes, because mocapping hands is difficult. And with keyframes animation, much like modelling geometry traditionally vs photogrammetry, we can't produce the needed amount of details in animations for them to look real. Moreover, we are extremely good at spotting something off with hands movement.
drunkenmagician 5y ago
Mouth movements were not 100% either. FWIW mouth animation is actually quite hard to get right, I remember artists at a previous company I worked at (also doing super realistic digital humans) spending hours try to get it just right, but still always seem to be able notice it being a bit off...

torginus 5y ago

I'm kinda confused why Unity keeps doing this - they keep putting out high-end demo after high-end demo, but that's not where there core userbase is. Their main users are people who build games for phones and indies, with basically zero usage in the AAA space. And Unity's performance/stability is still not that great afaik. It seems to me that they are trying to prove that they are a serious 'AAA' engine, but these demos aren't that convincing to me - AAA is a lot more than putting fancy shaders on high-poly models, it's about handling huge amounts of objects in a dynamics situation, displaying large worlds via streaming, having a workflow that accommodates every creative professional, and offering great performance and visuals even on very complex scenes. I've hear that even these highly impressive demos are fake - they built a ton of custom code for each one they rebuild core Unity features, meaning if you wanted to replicate this for yourself, you'd be in for a ton of development. Comparatively UE5's Nanite demo showed off a tech that's ready to go for production.

belval 5y ago
Pretty sure they want to break out of the core userbase. I once interviewed there and they really stressed that they were a 3D platform and not just a game engine. In that context it would make sense to make demos for the capabilities that people don't know that you have.
mrtweetyhack 5y ago
Etheryte 5y ago
I don't work in the industry, but as an outsider your question seems to answer itself. I would wager that Unity wants to take the step from the hobbyist space to the professional space — because surely that's where the bigger sums are made. Even if they're not there right now, consistently putting out content that indicates that you're working on it is a great way to captivate your audience over time and shift the perception of what your brand is about. In the startup space, this would be similar to building in the open — it helps you signal your brand, build a reputation and, hopefully, build a customer base that matches your sell.
5y ago

gigatexal 5y ago

Nope. Nope. Nope. Do not want. That’s too good. It’s creepy good. I need to know what’s real and what isn’t. This is so real that I’m freaking out a bit.

bhedgeoser 5y ago
She wasn't breathing.

playpause 5y ago

How come the latest cutting edge tech demos showcasing photorealism are barely more ‘realistic’ than a PS4 video game cutscene from a few years ago? There wasn’t any part of the demo video where the guy looked or moved like a human.

CanSpice 5y ago
Those cutscenes weren't rendered in real-time, instead taking longer than one second to render one second of video. This one is rendered in real-time. It's impressive for sure to render lifelike humans, but to do so in real-time is even more impressive.
- jay_kyburz 5y ago
  Yes, but that's the point. The should have released it as an App that we can interact with, pan around, move the chess pieces. It might as well just be a video. Actually they should have plugged in stockfish and let us play against their character.
- whywhywhywhy 5y ago
  > Those cutscenes weren't rendered in real-time Don't most games use realtime cutscenes these days? Nothing shown here is a gigantic leap to what you see in Death Stranding for example. I'm not looking at her and seeing an actress it still just looks like a very good 3D model.

davidhyde 5y ago

It does not look like they simulate the muscles underneath the skin which makes it feel uncanny unfortunately. Close though! Very close and impressive indeed. Static rendering is excellent.

farzher 5y ago

newobj 5y ago

Nope, still the uncanny valley

cercatrova 5y ago

Interesting comment from a game developer on /r/pcgaming [0] > The problem with Unity cinematics is that they are much less representative of their tools and engine out-of-the-box capabilities than Unreal's demos. > They use heavily customized assets specifically for a video and even do a lot of custom changes to the actual engine held by hot glue and duct tape to make it work. Then you try to recreate only fragments of what was show and nothing works. Even their released demos are usually not very portable and useful for devs, while in Unreal you can pretty much copy useful modules and most of it just works. This is why devs struggle to make things that look like demos Unity released 5 years ago... > Unreal is less smoke and mirror. They focus more on things that are actually available inside the editor (eg. the main character in Matrix Awakens was taken straight from the metahuman editor instead of spending hundreds of thousands of dollars on custom scanned model rigged by a huge team of world experts). > Unity does much better job with their demos to impress general public, but look at comments on their own forums and you will see over and over again how many devs are frustrated with their marketing tricks that don't turn into something real. > Some devs say that this difference comes from the fact that Unreal guys have to eat their own food that they make, because they also make games (like Fortnite), while Unity is more detached from actual needs of game devs and recently they became obsessed with film industry, even acquired Weta Digital that made LOTR and Avatar. It should be exciting and impressive, but the comments from their licensees (waiting for years to fix core issues) were quite negative. [0] http://www.reddit.com/r/pcgaming/comments/tjasoe/enemies_realtime_tech_demo_by_unity/i1jcyjr http://www.reddit.com/r/pcgaming/comments/tjasoe/enemies_rea...

deleted 5y ago
[deleted]
squeegmeister 5y ago
The "glued together" may be a little overstated. Their "Mega-City" demoed in 2019 is available to download and run as a standard unity project. I haven't looked at any of the code, but the fact that it runs on a mainline version of unity and not some forked, hacked together version should count for something.

ourmandave 5y ago

Demo released just in time for the Game Developers Conference, which they'll be doing a session at on March 24th. https://schedule.gdconf.com/session/inside-unitys-new-flagship-demo-presented-by-unity/887041 https://schedule.gdconf.com/session/inside-unitys-new-flagsh...

slackstation 5y ago

The eyes still look dead. All the raytracing in the world doesn't matter if the eyes don't look like they have life.

throwoutway 5y ago

I want to make a demo like this but I don’t even know where to start. Does it take thousands of hours of skills and training to become this good at demos?

jacquesm 5y ago

The illusion was pretty good until she started talking and then it fell apart completely for me.

ehsankia 5y ago

Am I the only one who finds that lipsync is ALWAYS off in every single hyper-realistic game / demo? Is lipsync also subject to uncanny valley or what? Why is it so damn hard to sync audio with lips in these simulations? Go to 1:34 in the video, slow it down to 0.25x on Youtube, and press play.

erikbye 5y ago
Lip syncing in games continues to disappoint. LA Noire from 2011 had some artifacts, but it's still better than most games since.

patientplatypus 5y ago

mr-pink 5y ago

what's the point of this? they had hyper realistic humans back in Myst back in 1993

yahn00 5y ago

I can find hyper-realistic humans anywhere, why waste computing power on this?

batch12 5y ago

Creepy. Most of these human renderings look dead to me.

stevev 5y ago

That’s great. Other studios are working on realistic problems and performance as well like Pearl Abyss. Difference is, they actually have games using the tech.

tw20212021 5y ago

What a stupid cookie popup. So a cookie from 'livecasinocomparer.com' is a Strictly Necessary Cookie. WTH

sharkster711 5y ago

Isn't that Carrie-Anne Moss?

alkonaut 5y ago

In-engine is not in-game. I'm not that interested in what a game engine can do unless it's also constrained by doing all the other things a game engine can do. Show me what's possible in unity at 100fps on a consumer system while doing normal game loop things as well.

shetill 5y ago

yet another project that will stay in preview forever

deleted 5y ago

[deleted]

aniijbod 5y ago

Most of the comments that I can see below are restricted to discussing the impact of announcement and demo on the Unity vs. Unreal situation. However, it turns out that here are other very relevant and important issues which arise from the technological specifics behind the demo which, although they obviously directly impinge on the rivalry, have potentially broader implications. 'Uncanny valley' issues related to face-based (as well as body motion-based) realism in simulation for both games development and video production, whilst there have been recent very significant advances in addressing them are still a (pun unavoidable) rapidly moving target, a target where this demo highlights both substantial advances and still as-yet unresolved serious shortcomings. The advances mostly relate to the introduction of machine learning and 15TB of training data into the content production workflow to reduce a rendering job from 6 hours per 50 frames to 3 milliseconds per frame as detailed in this article: https://www.globenewswire.com/news-release/2021/12/10/2349777/0/en/Senua-s-Saga-Hellblade-II-Gameplay-Reveals-Character-by-Ziva-Dynamics.html https://www.globenewswire.com/news-release/2021/12/10/234977... Returning for just a moment to the UE vs Unreal thing, those results were discussed in an article about a demo created using Ziva Dynamics's Real Time Trainer and Unreal Engine 4.26, but Ziva was subsequently acquired by Unity. The discussion I'd be interested to see here (or somewhere else!) is about how ZRT compares with (or perhaps even fits into?) Unreal Engine's Metahuman workflow and resulting content creation and realism results. Perhaps more importantly, how might the ML-driven capabilities of ZRT (will they still continue to be available under Unreal as well as Unity?) and ML-based impact upon realism be expected to manifest themselves Here's a link to the ZRT 'preparation workflow', which a Z-brush user who was trialling the trainer setup says took them about an hour to produce a workable result: https://drive.google.com/uc?export=download&id=15J6vX-6vU5EUuUDhvfrfDb-ZtoDfwcqg https://drive.google.com/uc?export=download&id=15J6vX-6vU5EU...

chakintosh 5y ago

Not as uncanny valley as UE's Metahumans.

globular-toast 5y ago

I've noticed that the more "realistic" animated humans become, the less conventionally attractive they become. I began to notice this years ago when characters started to have skin blemishes etc. It struck me as a simple gimmick then. Like all they could do was increase the texture resolution and to show that off you need "interesting" facial features. All of this is just trickery to distract you from the uncanny valley. The real test of this rendering is "perfect" faces like Charlize Theron or George Clooney. Those are real people too, by the way. I also notice that the latest "hyper-realistic" models are almost always female too. I wonder why that is.

sinuhe69 5y ago

From afar, the eyes seem to bulge out and up-close the hairs and hairs movement are still quite artificial, though.