5 ms·
That is a very impressive demo. Does anyone know if the details of the codec, and the measurements used by the headset to generate the facial stream, have been
by punkiln 3y ago
That is a very impressive demo. Does anyone know if the details of the codec, and the measurements used by the headset to generate the facial stream, have been publicly released yet?
The interview gets quite interesting nearer the end where Zuckerberg talks about his vision for the future, like training AI-backed avatars to simulate real people. Which apparently is not far off from reality - they joke about Fridman doing a podcast where he interviews an AI version of himself in the near future. Reminds me a bit of that Black Mirror episode where Miley Cyrus' character has AI versions of herself contained, or trapped, within consumer electronics.
It also occurs to me that this could be a useful technology for people with social anxiety who have to endure uncontrollable blushing when conversing in real life, or people with awkward facial tics. These could be filtered out or not even recorded before being encoded for transmission, while still giving a realistic appearance, making for a more comfortable presentation of the real-virtual self.
- zonkerdonker 3y agoAn entire fully "rigged" version of your face is generated from thousands of high resolution photos using a variety of facial expressions, distilled down into a model. Data from eye tracking and the cameras on the HMD are used to estimate the position of your face, lips, eyes etc, which is then fed into the pregenerated rigged model to be rendered.
- dharma1 3y agocameras on the Quest enough to capture all the facial/eye positions to drive the rigged facial model? impressive. They should open up the Codec avatar creation - lots of people have the hardware and the time/expertise to create them
- zonkerdonker 3y agoYou might think that, until you see the insane rig that's used to capture these scans: https://www.uploadvr.com/meta-codec-avatars-iphone-scan/ https://www.uploadvr.com/meta-codec-avatars-iphone-scan/ Each scan captures dozens of terabytes of data and lasts for hours, at least for the current high resolution avatars
- tempaccount420 3y agoI don't see why an iPhone 15 Pro wouldn't be able to capture these scans, especially with the new "spatial video" feature, which takes a "3d" video using multiple lenses.
- dharma1 3y agoYou'll get decent results but it won't be as good as with dSLRs and polarised studio light - if you want super detailed textures, be able to relight them etc
- dharma1 3y agothat's a full Debevec style light stage but you can do this stuff with a couple of dSLRs and a home lighting setup with polarised light - kind of standard photogrammetry/hq texture/normal map generation techniques. You can see from the normal map in the video it's pretty detailed but at least for a single face capture you can do this at home. I'm not sure what secret sauce they have for capturing multiple facial expressions and some ML magic how to morph/animate between those. Btw you can get good results with things like https://www.unrealengine.com/en-US/metahuman https://www.unrealengine.com/en-US/metahuman too
- JohnFen 3y ago> Zuckerberg talks about his vision for the future, like training AI-backed avatars to simulate real people This vision of the future sounds like a nightmare to me.
- deleted 3y ago[deleted]
- latency-guy2 3y agoDo you get scared by video games too?
- deleted 3y ago[deleted]
- Keyframe 3y agoI wouldn't be surprised if it's some form of (a variant or advancement) of FACS which we've used in animation eons ago https://en.wikipedia.org/wiki/Facial_Action_Coding_System https://en.wikipedia.org/wiki/Facial_Action_Coding_System Basically for each of the identified poses you have a key and you grab a pose of it (photogrammetry here) and then you would capture performance and identify somehow which combination of keys and weights would be translated to you model. Sounds easy but it ain't.