10 ms·

Oculus Rift hack transfers your facial expressions onto your virtual avatar

91 points by 51Cards 11y ago

tonydiv 11y ago
This is a fundamental technology that will be core to social VR. I really hope they nail it.
- moron4hire 11y ago
  Even something as simple as head gestures, which can be done with just about every currently-known HMD, is hugely helpful in expressing emotion. Check out AltspaceVR, it's pretty incredible. I had an hour-long conversation with someone and even after the first 5 minutes it felt like we were just sitting in Halloween costumes, talking to each other.
  - tonydiv 11y ago
    I have tried Altspace many times :) We've built something similar in some regards. We are helping people learn foreign languages in immersive environments (including VR): http://learnimmersive.com http://learnimmersive.com
- haoli81 11y ago
  Hi, i m the lead of this research project which was really more of a first prototype and feasibility test. We have already started working on the next version though, but can't comment on it yet.
egypturnash 11y ago
One step closer to the Metaverse.
- jnem 11y ago
  If I were William Gibson, or even Neal Stephenson, I'd be sitting back right now saying, "yup. Told ya".
  - drzaiusapelord 11y ago
    Except they've been wrong almost universally in their career from a predicative POV (which is a bad way to judge sci-fi, btw. They write dramatic narratives, theyre not futurists trying to predict future tech, they're storytellers). The reality is we don't know if this stuff will ever catch on. Mounting a big black box on your face is limiting and unappealing to most. Hardcore gamers? Probably will deal with it. Everyone else? Probably not. There's also a fundamental issue here that's rarely addressed: that putting meatspace conventions into virtual spaces rarely works out. I don't want to have to wake up, dress, shower, etc my avatar so she can get in a car and drive to the virtual mall to walk into the virtual store to buy stuff. I just want to load up amazon.com and scroll to the product I want and one-click buy it. We tried Second Life, Playstation Home, etc and we didn't like them. Adding whiz-bang 3D isn't going to help a largely failed and discredited concept. The meta-verse is just annoyingly tedious and time-wasting. Those 80s visionaries had no conception of what things like the web or mobile could be. They didn't realize how powerful those technologies are (Hi, we're using it right now to communicate instead of Second Life for a reason). So they projected this idea of "just put people in computers" which helps a dramatic narrative as its easy to understand but isn't very practical in real life.
    - jkaunisv1 11y ago
      You're not wrong, but you seem to be missing a huge part of the picture. Mounting a big black box sucks, yes, but the tech will mature, miniaturize and become more appealing. Even before then, I think many a CEO would rather put on a black box to have a meeting rather than fly for 16 hours. It's not all about putting meatspace conventions into virtual spaces. The appeal of the metaverse isn't about having to walk around a digital world to get anything done. It's about being able to connect with people across vast physical spaces with a lot of the same quality of the connection as in-person. And more generally, about providing that quality of experience in a ton of contexts. When buying something online, I think a lot of people would value being able to look at the product from various angles before buying. I know I've cancelled orders because I wasn't sure how a shoe would look based on the one or two angles a site provided. And yes, obviously, you will probably still buy toilet paper and books on amazon just by scrolling in your browser and clicking. But other purchasing processes will be hugely enhanced by VR. You could better gauge the fit (and when graphics improve, texture/fabric) of clothing. IKEA can go to the next level and actually project the furniture you want into a 3d mockup of the room you want it to go in. Hell, you could probably fly a special drone around your room and it would create the model for you. Yes, Second Life wasn't everything people imagined. But it was a crude first stab at the problem. Even Oculus is still relatively crude, but it's very immersive already and I don't see why it won't eventually become very refined and convincing. Look at the 3D rendering we're capable of already in games and movies, throw in some Moore's Law and wait to see what comes out.
- haoli81 11y ago
  Indeed. This is our goal.
zyxley 11y ago
It would be really interesting to see something like this mixed with commercial software like FaceRig (http://store.steampowered.com/app/274920/ http://store.steampowered.com/app/274920/).
dvirsky 11y ago
So when can we have our OASIS accounts and dystopian future?
ahamino 11y ago
There are a lot of players who are trying to commercialize facial expression tracking. Notably, http://affectiva.com http://affectiva.com https://www.youtube.com/watch?v=AbO0Po8Mgis https://www.youtube.com/watch?v=AbO0Po8Mgis disclaimer: I work for them.
munro 11y ago
Awesome, this research project looks good to go! I could definitely see this technology being integrated into VR headsets. The strain gauges could come bundled, and the lower half camera could be built into the bottom, and engineered to fit more flush. Only potential stopper sounds like it requires lot of computation power to measure the face.
entwife 11y ago
Is it necessary to have the screen/VR features of the Occulus Rift to do the facial expression recognition? Or, is data from something more lightweight, e.g. Google Glass, adequate to do facial recognition? Looking forward to better virtual meetings, less flying.
- philtar 11y ago
  The facial recognition is from the Intel RealSense. Read the article.
fsimoneschi2 11y ago
Have a look at https://www.kickstarter.com/projects/fove/fove-the-worlds-first-eye-tracking-virtual-reality https://www.kickstarter.com/projects/fove/fove-the-worlds-fi... They are working on a similar technology.
- kaitynotes 11y ago
  Eye tracking looks very promising. After seeing the video, I'm curious to see how they plan on adding facial expression tracking.
  - aberrantwolf 11y ago
    I think face tracking is planned for the version after the Kickstarter.
    - fsimoneschi2 11y ago
      They will probably add some IR or optical camera beneath the headset. Not quite sure if they can also pick up some face expression by using the eye-camera as well (like smiling).

91 points by 51Cards 11y ago

tonydiv 11y ago

This is a fundamental technology that will be core to social VR. I really hope they nail it.

moron4hire 11y ago
Even something as simple as head gestures, which can be done with just about every currently-known HMD, is hugely helpful in expressing emotion. Check out AltspaceVR, it's pretty incredible. I had an hour-long conversation with someone and even after the first 5 minutes it felt like we were just sitting in Halloween costumes, talking to each other.
- tonydiv 11y ago
  I have tried Altspace many times :) We've built something similar in some regards. We are helping people learn foreign languages in immersive environments (including VR): http://learnimmersive.com http://learnimmersive.com
haoli81 11y ago
Hi, i m the lead of this research project which was really more of a first prototype and feasibility test. We have already started working on the next version though, but can't comment on it yet.

egypturnash 11y ago

One step closer to the Metaverse.

jnem 11y ago
If I were William Gibson, or even Neal Stephenson, I'd be sitting back right now saying, "yup. Told ya".
- drzaiusapelord 11y ago
  Except they've been wrong almost universally in their career from a predicative POV (which is a bad way to judge sci-fi, btw. They write dramatic narratives, theyre not futurists trying to predict future tech, they're storytellers). The reality is we don't know if this stuff will ever catch on. Mounting a big black box on your face is limiting and unappealing to most. Hardcore gamers? Probably will deal with it. Everyone else? Probably not. There's also a fundamental issue here that's rarely addressed: that putting meatspace conventions into virtual spaces rarely works out. I don't want to have to wake up, dress, shower, etc my avatar so she can get in a car and drive to the virtual mall to walk into the virtual store to buy stuff. I just want to load up amazon.com and scroll to the product I want and one-click buy it. We tried Second Life, Playstation Home, etc and we didn't like them. Adding whiz-bang 3D isn't going to help a largely failed and discredited concept. The meta-verse is just annoyingly tedious and time-wasting. Those 80s visionaries had no conception of what things like the web or mobile could be. They didn't realize how powerful those technologies are (Hi, we're using it right now to communicate instead of Second Life for a reason). So they projected this idea of "just put people in computers" which helps a dramatic narrative as its easy to understand but isn't very practical in real life.
  - jkaunisv1 11y ago
    You're not wrong, but you seem to be missing a huge part of the picture. Mounting a big black box sucks, yes, but the tech will mature, miniaturize and become more appealing. Even before then, I think many a CEO would rather put on a black box to have a meeting rather than fly for 16 hours. It's not all about putting meatspace conventions into virtual spaces. The appeal of the metaverse isn't about having to walk around a digital world to get anything done. It's about being able to connect with people across vast physical spaces with a lot of the same quality of the connection as in-person. And more generally, about providing that quality of experience in a ton of contexts. When buying something online, I think a lot of people would value being able to look at the product from various angles before buying. I know I've cancelled orders because I wasn't sure how a shoe would look based on the one or two angles a site provided. And yes, obviously, you will probably still buy toilet paper and books on amazon just by scrolling in your browser and clicking. But other purchasing processes will be hugely enhanced by VR. You could better gauge the fit (and when graphics improve, texture/fabric) of clothing. IKEA can go to the next level and actually project the furniture you want into a 3d mockup of the room you want it to go in. Hell, you could probably fly a special drone around your room and it would create the model for you. Yes, Second Life wasn't everything people imagined. But it was a crude first stab at the problem. Even Oculus is still relatively crude, but it's very immersive already and I don't see why it won't eventually become very refined and convincing. Look at the 3D rendering we're capable of already in games and movies, throw in some Moore's Law and wait to see what comes out.
haoli81 11y ago
Indeed. This is our goal.

zyxley 11y ago

It would be really interesting to see something like this mixed with commercial software like FaceRig (http://store.steampowered.com/app/274920/ http://store.steampowered.com/app/274920/).

dvirsky 11y ago

So when can we have our OASIS accounts and dystopian future?

ahamino 11y ago

There are a lot of players who are trying to commercialize facial expression tracking. Notably, http://affectiva.com http://affectiva.com https://www.youtube.com/watch?v=AbO0Po8Mgis https://www.youtube.com/watch?v=AbO0Po8Mgis disclaimer: I work for them.

munro 11y ago

Awesome, this research project looks good to go! I could definitely see this technology being integrated into VR headsets. The strain gauges could come bundled, and the lower half camera could be built into the bottom, and engineered to fit more flush. Only potential stopper sounds like it requires lot of computation power to measure the face.

entwife 11y ago

Is it necessary to have the screen/VR features of the Occulus Rift to do the facial expression recognition? Or, is data from something more lightweight, e.g. Google Glass, adequate to do facial recognition? Looking forward to better virtual meetings, less flying.

philtar 11y ago
The facial recognition is from the Intel RealSense. Read the article.

fsimoneschi2 11y ago

Have a look at https://www.kickstarter.com/projects/fove/fove-the-worlds-first-eye-tracking-virtual-reality https://www.kickstarter.com/projects/fove/fove-the-worlds-fi... They are working on a similar technology.

kaitynotes 11y ago
Eye tracking looks very promising. After seeing the video, I'm curious to see how they plan on adding facial expression tracking.
- aberrantwolf 11y ago
  I think face tracking is planned for the version after the Kickstarter.
  - fsimoneschi2 11y ago
    They will probably add some IR or optical camera beneath the headset. Not quite sure if they can also pick up some face expression by using the eye-camera as well (like smiling).