6 ms·

How did they go from second-life knockoffs to this? This is actually impressive, and while not perfect, is close enough that they'll probably reach a version t

How did they go from second-life knockoffs to this? This is actually impressive, and while not perfect, is close enough that they'll probably reach a version that is viable very soon.

gfodor 3y ago
They didn't, people just haven't been paying attention. They've been working on these avatars for almost a decade in parallel waiting for the hardware to catch up. The cartoon avatars are a function of hardware limitations.
capableweb 3y ago
> How did they go from second-life knockoffs to this? Typically you do one MVP version and as soon as you've knocked that out, start working on a proper version, integrating feedback from the MVP as it comes in. I'm guessing they always wanted to have photorealism, but realized it'd take a while to get in place so they launched a quick version then got back to work.
- DoingIsLearning 3y ago
  > it'd take a while to get in place so they launched a quick version then got back to work. This is not a phone app. Physical demos that propose something that a user has no point of comparison with (new paradigm) absolutely need to be on either end of the maturity scale. - Either very raw barebone MPVs that should be only for lab/research aware people who won't cling to details and are able to see the 'principle' of what it could be. - Or absolutely imaculate picture perfect marketing demos that you can show to non-technical people Anything in between is a recipe for getting shot down, getting no buy-in, and receiving no insightful feedback.
  - capableweb 3y ago
    > This is not a phone app. Really? Definitely looks like something a simple phone should be able to process... Are they not wearing phones on their heads in the demo?! This "Publish > Feedback > Iterate > Repeat" loop doesn't just apply to smartphone apps, you can basically do this with anything in life, from hardware to mixology. I guess you could say the same thing about LLMs, since the initial models were terrible and just basically demos, they shouldn't actually have published those at all?
    - DoingIsLearning 3y ago
      > This "Publish > Feedback > Iterate > Repeat" loop doesn't just apply to smartphone apps, you can basically do this with anything in life, from hardware to mixology I again disagree. "Publish > Feedback > Iterate > Repeat" loop works when users can build on it from a previous reality that they can compare against. Physical demos that propose something that a user has no point of comparison with (new paradigm) you can either show a sketch/concept of what you want to do (to a tecnhical audience). Or build a semi-realistic demo to a non technical audience. Anything in between will probably be met with confusion or people focusing on details that are not the concept that you are showcasing.
  - codetrotter 3y ago
    > Anything in between is a recipe for getting shot down, getting no buy-in, and receiving no insightful feedback. I don’t agree. I dislike Fb to the extent that I already deleted my account quite a while ago. And I thought the Meta attempt at a meta verse with cartoon characters looked ridiculous. But the video in the OP link, with the photo real avatars. This is a whole other level. It’s super cool. The fact that they made a ridiculous cartoon Metaverse first does not detract from the coolness of this photorealistic thing. Still, I’m not gonna use any Metaverse that Meta will make. Photoreal or otherwise. But that is because I dislike Meta as a company. The point is that what they have made here is some top notch stuff. I look forward to seeing Apple compete with this. I hope that with an iPhone and an Apple VR headset I can get to experience something similar in a few years.
    - DoingIsLearning 3y ago
      > The fact that they made a ridiculous cartoon Metaverse first does not detract from the coolness of this photorealistic thing. I agree with you on the coolness of this demo. I disagree that the cartoon demo does not detract. I suspect a large majority of users and advertisers assumed Zuckerberg went off-piste completely. This interview with Lex is much more aligned with what should have been their first public demo of a Metaverse 'vision'.
      - kyleee 3y ago
        A negligible amount of people have an attention span and memory as strong as you suggest is commonplace, so I think your point is moot.
        thfuran 3y ago
        Have you never heard of the importance of first impressions?
rngname22 3y ago
The second-life knockoffs were only necessary because of the trade-off they made to target low-price HMD market and on-device GPU+CPU processing rather than PC VR or processing puck design. This photogrammetry based avatar stuff has already been working for years. See Meta's previous "Codec Avatar" in 2021 (https://www.youtube.com/watch?v=bS4Gf0PWmZs https://www.youtube.com/watch?v=bS4Gf0PWmZs) or Unreal's MetaHuman (https://www.roadtovr.com/epic-metahuman-mesh-import-scan-real-people-digital-human-avatar/ https://www.roadtovr.com/epic-metahuman-mesh-import-scan-rea...).
- dharma1 3y ago
  what's the score with Epic and Apple Vision btw? I saw Lumen and Nanite now work in UE 5.3 on M2 silicon but not sure if it will be possible to develop apps for Apple Vision with Unreal, they just mention Unity? edit: looks like Epic is working on support for VisionOS in UE5 https://github.com/EpicGames/UnrealEngine/tree/ue5-main/Engine/Platforms/VisionOS/Source https://github.com/EpicGames/UnrealEngine/tree/ue5-main/Engi...
IKantRead 3y ago
> is close enough that they'll probably reach a version that is viable very soon. Haven't we seen this type of thinking to be proven a fallacy enough in the last decade to at least be skeptical of solving that last 1% until we actually see it?
nitwit005 3y ago
The background is either a white or black void, so they removed the entire "metaverse" from it, which I'm sure fixes most performance issues. Rendering two fairly realistic models on the screen at once isn't a big challenge these days, if they're the only thing being rendered.
- sangnoir 3y ago
  > The background is either a white or black void, so they removed the entire "metaverse" from it, which I'm sure fixes most performance issues. I doubt it's a performance issue: the 3D model of a rectilinear room has fewer vertices than used to represent a single earlobe. It most likely is just an unimplemented feature; they also could have "cheated" by chroma-keying a 2D-texture background or rotoscoped the avatars onto a another 3D room scene, but what's the point when its just talking heads?
  - nitwit005 3y ago
    Surely people have higher expectations for the environment their avatar is going to be in than a literal box? Games often only have the highest detail models present in cutscenes, where the rest of the game can be limited, and the camera can be controlled to limit how many characters are visible.
    - sangnoir 3y ago
      > Surely people have higher expectations for the environment their avatar is going to be in than a literal box? Yes - they do, but you don't have to get fancy with the environment for 2 static avatars locked in place. What are the odds that rendering a room was dropped because it is allegedly GPU-intensive vs it being a feature that is not fully-baked for a public debut yet?
  - wryanzimmerman 3y ago
    I think a large part of the black background choice was to hide the missing bodies; any light past chest level is going to push this into uncanny valley pretty fast because the modeling ends at the chest like talking Greek busts. Totally agree that it’s a stylistic and presentational decision and not a computational limitation.
bookaway 3y ago
I'm not sure how much facebook paid Fridman for this podcast but whatever it was, whether it was 0 or 10 million, it's probably the best value for money PR they've done in the last decade...or ever?
- deleted 3y ago
  [deleted]
- baby 3y ago
  Why would they pay Lex for this? It's basically his full time job to do interviews for his podcast. That demo was going to happen one day or another as the tech progresses, and it'll only get better from there.
  - bookaway 3y ago
    The implication in my comment is that they might indeed have not paid him anything. And Fridman on principle may not have accepted payment even if offered. But who knows what type of deals go down behind the curtain in the podcasting world, it’s not unheard of to come across interviews that look like paid promotion where the podcaster doesn’t explicitly state it as such. But my point was more really that even if they paid him 10 million, the PR would have been worth it. >That demo was going to happen one day or another as the tech progresses Yes, all amazing technology is going to happen eventually — no one really cares about that. They care about seeing it happen in the present, right in front of their eyes. The metaverse was mocked profusely in 2023, and we have gone through multiple cycles of “This is going to be the year of VR” in past decades. Tech stocks have been flat the last couple of months and Meta is a public company. No one on the outside would have batted an eye if they quietly shut this whole thing down after being so relentlessly mocked for the past year. So saying Zuckerberg doing this demo was inevitable after the fact is a bit flippant I think. And if facebook did a demo with a bunch of celebrities and put it on their website I’m not sure it would have the same effect as this long interview with Fridman. It wouldn’t have the same effect on me that’s for sure. And I’m pretty sure they would have paid those celebrities.
    - 3y ago
rapjr9 3y ago
It is definitely not perfect. It doesn't seen to track distortions of the skin on the face, such as around the mouth on the cheeks. It tracks eye blinks, but does it track pupil dilation? Doesn't seem like it could track forehead creases, though possibly. Do the avatar's eyebrows track the real eyebrows? No way to tell, but an important part of expression. I rarely see the eyes moving, maybe an artifact of being in a dark space with only one thing to look at? Pursing of the lips is not tracked. Tongue? I don't see shoulders moving. It's an impressive demo, yet as a repeated life experience it probably would start to feel artificial and uncanny. The video does discuss some of this. I get the impression some of it is prediction and some of it is tracking.
- gumballindie 3y ago
  I doubt casual users care about these.
  - jwestbury 3y ago
    Casual users probably care the most, though perhaps not consciously. But all of these things are critical for understanding body language and facial expressions, and without those this is just a nice tech demo.
- wbm1 3y ago
  All things that will come with time. What makes for a viable product is if the mind is 'tricked' enough into believing it's seeing an actual person instead of a 3D replica.
fsociety 3y ago
That’s the thing. This whole metaverse push has been in the works for a long time. All that ad money being funneled into R&D. It does not mean they will make a killer product. But behind closed doors they have some amazing tech.
iAkashPaul 3y ago
Nvidia had built Maxine back in Oct of 2020. Arun Mallya also published One-shot Talking Head Synthesis in 2021 which does the same thing except removing the background from the rendering

How did they go from second-life knockoffs to this? This is actually impressive, and while not perfect, is close enough that they'll probably reach a version that is viable very soon.

gfodor 3y ago

They didn't, people just haven't been paying attention. They've been working on these avatars for almost a decade in parallel waiting for the hardware to catch up. The cartoon avatars are a function of hardware limitations.

capableweb 3y ago

> How did they go from second-life knockoffs to this? Typically you do one MVP version and as soon as you've knocked that out, start working on a proper version, integrating feedback from the MVP as it comes in. I'm guessing they always wanted to have photorealism, but realized it'd take a while to get in place so they launched a quick version then got back to work.

DoingIsLearning 3y ago
> it'd take a while to get in place so they launched a quick version then got back to work. This is not a phone app. Physical demos that propose something that a user has no point of comparison with (new paradigm) absolutely need to be on either end of the maturity scale. - Either very raw barebone MPVs that should be only for lab/research aware people who won't cling to details and are able to see the 'principle' of what it could be. - Or absolutely imaculate picture perfect marketing demos that you can show to non-technical people Anything in between is a recipe for getting shot down, getting no buy-in, and receiving no insightful feedback.
- capableweb 3y ago
  > This is not a phone app. Really? Definitely looks like something a simple phone should be able to process... Are they not wearing phones on their heads in the demo?! This "Publish > Feedback > Iterate > Repeat" loop doesn't just apply to smartphone apps, you can basically do this with anything in life, from hardware to mixology. I guess you could say the same thing about LLMs, since the initial models were terrible and just basically demos, they shouldn't actually have published those at all?
  - DoingIsLearning 3y ago
    > This "Publish > Feedback > Iterate > Repeat" loop doesn't just apply to smartphone apps, you can basically do this with anything in life, from hardware to mixology I again disagree. "Publish > Feedback > Iterate > Repeat" loop works when users can build on it from a previous reality that they can compare against. Physical demos that propose something that a user has no point of comparison with (new paradigm) you can either show a sketch/concept of what you want to do (to a tecnhical audience). Or build a semi-realistic demo to a non technical audience. Anything in between will probably be met with confusion or people focusing on details that are not the concept that you are showcasing.
- codetrotter 3y ago
  > Anything in between is a recipe for getting shot down, getting no buy-in, and receiving no insightful feedback. I don’t agree. I dislike Fb to the extent that I already deleted my account quite a while ago. And I thought the Meta attempt at a meta verse with cartoon characters looked ridiculous. But the video in the OP link, with the photo real avatars. This is a whole other level. It’s super cool. The fact that they made a ridiculous cartoon Metaverse first does not detract from the coolness of this photorealistic thing. Still, I’m not gonna use any Metaverse that Meta will make. Photoreal or otherwise. But that is because I dislike Meta as a company. The point is that what they have made here is some top notch stuff. I look forward to seeing Apple compete with this. I hope that with an iPhone and an Apple VR headset I can get to experience something similar in a few years.
  - DoingIsLearning 3y ago
    > The fact that they made a ridiculous cartoon Metaverse first does not detract from the coolness of this photorealistic thing. I agree with you on the coolness of this demo. I disagree that the cartoon demo does not detract. I suspect a large majority of users and advertisers assumed Zuckerberg went off-piste completely. This interview with Lex is much more aligned with what should have been their first public demo of a Metaverse 'vision'.
    - kyleee 3y ago
      A negligible amount of people have an attention span and memory as strong as you suggest is commonplace, so I think your point is moot.
      - thfuran 3y ago
        Have you never heard of the importance of first impressions?

rngname22 3y ago

The second-life knockoffs were only necessary because of the trade-off they made to target low-price HMD market and on-device GPU+CPU processing rather than PC VR or processing puck design. This photogrammetry based avatar stuff has already been working for years. See Meta's previous "Codec Avatar" in 2021 (https://www.youtube.com/watch?v=bS4Gf0PWmZs https://www.youtube.com/watch?v=bS4Gf0PWmZs) or Unreal's MetaHuman (https://www.roadtovr.com/epic-metahuman-mesh-import-scan-real-people-digital-human-avatar/ https://www.roadtovr.com/epic-metahuman-mesh-import-scan-rea...).

dharma1 3y ago
what's the score with Epic and Apple Vision btw? I saw Lumen and Nanite now work in UE 5.3 on M2 silicon but not sure if it will be possible to develop apps for Apple Vision with Unreal, they just mention Unity? edit: looks like Epic is working on support for VisionOS in UE5 https://github.com/EpicGames/UnrealEngine/tree/ue5-main/Engine/Platforms/VisionOS/Source https://github.com/EpicGames/UnrealEngine/tree/ue5-main/Engi...

IKantRead 3y ago

> is close enough that they'll probably reach a version that is viable very soon. Haven't we seen this type of thinking to be proven a fallacy enough in the last decade to at least be skeptical of solving that last 1% until we actually see it?

nitwit005 3y ago

The background is either a white or black void, so they removed the entire "metaverse" from it, which I'm sure fixes most performance issues. Rendering two fairly realistic models on the screen at once isn't a big challenge these days, if they're the only thing being rendered.

sangnoir 3y ago
> The background is either a white or black void, so they removed the entire "metaverse" from it, which I'm sure fixes most performance issues. I doubt it's a performance issue: the 3D model of a rectilinear room has fewer vertices than used to represent a single earlobe. It most likely is just an unimplemented feature; they also could have "cheated" by chroma-keying a 2D-texture background or rotoscoped the avatars onto a another 3D room scene, but what's the point when its just talking heads?
- nitwit005 3y ago
  Surely people have higher expectations for the environment their avatar is going to be in than a literal box? Games often only have the highest detail models present in cutscenes, where the rest of the game can be limited, and the camera can be controlled to limit how many characters are visible.
  - sangnoir 3y ago
    > Surely people have higher expectations for the environment their avatar is going to be in than a literal box? Yes - they do, but you don't have to get fancy with the environment for 2 static avatars locked in place. What are the odds that rendering a room was dropped because it is allegedly GPU-intensive vs it being a feature that is not fully-baked for a public debut yet?
- wryanzimmerman 3y ago
  I think a large part of the black background choice was to hide the missing bodies; any light past chest level is going to push this into uncanny valley pretty fast because the modeling ends at the chest like talking Greek busts. Totally agree that it’s a stylistic and presentational decision and not a computational limitation.

bookaway 3y ago

I'm not sure how much facebook paid Fridman for this podcast but whatever it was, whether it was 0 or 10 million, it's probably the best value for money PR they've done in the last decade...or ever?

deleted 3y ago
[deleted]
baby 3y ago
Why would they pay Lex for this? It's basically his full time job to do interviews for his podcast. That demo was going to happen one day or another as the tech progresses, and it'll only get better from there.
- bookaway 3y ago
  The implication in my comment is that they might indeed have not paid him anything. And Fridman on principle may not have accepted payment even if offered. But who knows what type of deals go down behind the curtain in the podcasting world, it’s not unheard of to come across interviews that look like paid promotion where the podcaster doesn’t explicitly state it as such. But my point was more really that even if they paid him 10 million, the PR would have been worth it. >That demo was going to happen one day or another as the tech progresses Yes, all amazing technology is going to happen eventually — no one really cares about that. They care about seeing it happen in the present, right in front of their eyes. The metaverse was mocked profusely in 2023, and we have gone through multiple cycles of “This is going to be the year of VR” in past decades. Tech stocks have been flat the last couple of months and Meta is a public company. No one on the outside would have batted an eye if they quietly shut this whole thing down after being so relentlessly mocked for the past year. So saying Zuckerberg doing this demo was inevitable after the fact is a bit flippant I think. And if facebook did a demo with a bunch of celebrities and put it on their website I’m not sure it would have the same effect as this long interview with Fridman. It wouldn’t have the same effect on me that’s for sure. And I’m pretty sure they would have paid those celebrities.
  - 3y ago

rapjr9 3y ago

It is definitely not perfect. It doesn't seen to track distortions of the skin on the face, such as around the mouth on the cheeks. It tracks eye blinks, but does it track pupil dilation? Doesn't seem like it could track forehead creases, though possibly. Do the avatar's eyebrows track the real eyebrows? No way to tell, but an important part of expression. I rarely see the eyes moving, maybe an artifact of being in a dark space with only one thing to look at? Pursing of the lips is not tracked. Tongue? I don't see shoulders moving. It's an impressive demo, yet as a repeated life experience it probably would start to feel artificial and uncanny. The video does discuss some of this. I get the impression some of it is prediction and some of it is tracking.

gumballindie 3y ago
I doubt casual users care about these.
- jwestbury 3y ago
  Casual users probably care the most, though perhaps not consciously. But all of these things are critical for understanding body language and facial expressions, and without those this is just a nice tech demo.
wbm1 3y ago
All things that will come with time. What makes for a viable product is if the mind is 'tricked' enough into believing it's seeing an actual person instead of a 3D replica.

fsociety 3y ago

That’s the thing. This whole metaverse push has been in the works for a long time. All that ad money being funneled into R&D. It does not mean they will make a killer product. But behind closed doors they have some amazing tech.

iAkashPaul 3y ago

Nvidia had built Maxine back in Oct of 2020. Arun Mallya also published One-shot Talking Head Synthesis in 2021 which does the same thing except removing the background from the rendering