5 ms·

IPad 2 gets glasses-free 3D display using front-facing camera for head tracking

275 points by jashmenn 16y ago

deleted 16y ago
[deleted]
jerf 16y ago
See also http://www.youtube.com/watch?v=h5QSclrIdlE http://www.youtube.com/watch?v=h5QSclrIdlE , which is actually a product that you can buy, though I think it was never localized for the US. Edit: Also, http://johnnylee.net/projects/wii/ http://johnnylee.net/projects/wii/ , under "Head Tracking for Desktop VR Displays using the Wii Remote" (down a bit). (Also, to be clear, I just mean these as interesting "see alsos". I am not saying "it's been done" (so what?) or "it doesn't work" (I have no direct experience to make a judgment). Nor am I saying anyone accused me otherwise. The iPad is still a potential interesting sweet spot for this tech.)
- arrogant 16y ago
  http://kotaku.com/#!5541044/looksleys-line-up-micro+review-broken-promise http://kotaku.com/#!5541044/looksleys-line-up-micro+review-b... Looks like it did come out in the US, although from the review it looks like we might not have heard about it because the effect just didn't work well enough. Maybe this implementation will deliver? It looks fairly promising, and the effect looks like it might be more pronounced on the (compared to the DSi) large screen of the iPad. Of course, I'd like to pass judgement on it personally, so here's hoping for a demo we can hold in our hands soon.
  - kenjackson 16y ago
    This seems like a big deal even if it didn't work in the previous attempts. In part because I feel like it eventually will work. I'm excited and impressed by it. Although I do get a little disappointment that I didn't think of it. What the heck is my brain doing in my free time?!
    - jodrellblank 16y ago
      What the heck is my brain doing in my free time?! If you're bored... people ask for a touch sensitive rear surface - how about using the rear camera for gesture control? Wave your finger in front of it to scroll, that kind of thing. Also, some acoustic gesture recognition would be good to pick up one/two/three fingernail taps on the back.
      - kenjackson 16y ago
        Apple's already patented it. I don't want to get sued into oblivion. :-) http://www.addictivetips.com/mobile/iphone-control-through-camera-gestures-apple-patent-reveals/ http://www.addictivetips.com/mobile/iphone-control-through-c...
        jodrellblank 16y ago
        That's annoying, but it only seems to cover tap detection by accelerometer where I was thinking of tap detection by audio click, which definitely seems plausible from a quick test of the sound recorder. Also it only covers finger swipe detection/direction determining, where I was also imagining gestures from holding fingers a few centimeters away, such as scissoring two fingers, thumb/finger flick, two finger scroll movement, two fingertips touching, etc. A quick test with the iPhone camera suggests that might have to be too carefully positioned to be practical, but should be possible at some level.
    - jurjenh 16y ago
      I'm excited and impressed by it mainly because I did think of it and started implementing it, thinking to develop it as a desktop interface, or possibly a cad program interface. Lots of linear algebra involved... However, I stopped mainly because at the time (about 3-4 years ago) my webcam and opencv introduced a delay that was unacceptable and caused visual jarring in a small application (I think I was using a window of 320x240 or so). I've also seen a demo done in flash by MrDoob of Three.js fame (IIRC), so it's been done by several people. I haven't really seen a useful implementation yet, so this is getting more interesting with these smaller, higher spec integrated devices...
deleted 16y ago
[deleted]
Dramatize 16y ago
It's starting to feel like the future.
- rms 16y ago
  Yup. I'm wondering when more average people are going to start noticing this.
  - idonthack 16y ago
    I'm sure your father was saying the same thing forty years ago.
    - Dramatize 16y ago
      But this is what I was told the future would be when I was a kid.
MichaelApproved 16y ago
This could be extended with eye tracking to create even more impressive functions.
- Devilboy 16y ago
  I doubt the iPad's camera is good enough for eye tracking but maybe in future devices.
hook 16y ago
This is pretty amazing. I think that this coupled with multi-touch interactivity could be pretty revolutionary. It turns the iPad into an "iWindow".
- edw 16y ago
  Yes! As I was looking at the cube, I wanted the viewer to reach out and manipulate it: spin, pinch, what-have-you. I think the worst thing that could happen is that this sort of experience get baked into the core of the iPad's UI (or the UI of any tablet). But I could see imaging applications (MRIs?) where it would be valuable.
  - rflrob 16y ago
    I suspect that touching the screen would, at all but one depth plane, break the illusion. The brain is eerily good at fooling itself, until it receives contradictory information.
evo_9 16y ago
Is this more or less what the new nintendo handheld does? How do they differ if not?
- jarin 16y ago
  The 3DS screen is closer to one of those lenticular "3d" things you'd see on a ruler or a bookmark as a kid. It uses two interlaced images that are fed to each eye separately. The iPad 2 screen is just a normal screen. http://loot-ninja.com/2010/06/22/how-the-nintendo-3ds-screen-works/ http://loot-ninja.com/2010/06/22/how-the-nintendo-3ds-screen... Basically, the iPad 2 demo only gives the illusion of being 3D when you move it, but since your head and hands are pretty much constantly moving it should look "mostly 3D" if it has enough sensitivity.
  - evo_9 16y ago
    Thx, yeah I knew the ipad screen was a standard display, but having only seen ads for the new nintendo I wasn't sure what their tech was.
- arrogant 16y ago
  I'm pretty sure that the 3DS isn't doing any head tracking. If you move your head much at all the 3D effect breaks; you have to be looking practically dead at it.
  - lazerwalker 16y ago
    Yep. The way the 3D effect works, there are essentially physical barriers inside the screen separating the different views being rendered for each eye. Move your head so that you're not viewing the screen straight-on and the barriers don't work properly, ruining the effect.
franck 16y ago
I wonder what happens when several people try and look over your shoulder. How does it handle multiple heads?
- SammoJ 16y ago
  It could only render for a single viewpoint. I would imagine the simplest approach (i.e. without tracking) would be to render for the viewpoint of the closest detected head.
drivebyacct2 16y ago
So this is "3d" now? Wow, I've got a lot of cool stuff that is 3d, including a prototype compiz plugin that does roughly the same thing. Gives you spatial context for windows on your desktop environment via Kinect and OpenCL. I had no idea I could get away with calling this "3d", and have people in the comments thinking it's the same technology as the 3ds.
- arrogant 16y ago
  3D on a screen is always an illusion of some sort. The 3DS is just a different illusion (albeit a more technically involved one, for sure). It's not any more "3D" than this. Perspective not changing when your head moves is one of the things that breaks a 3D illusion; with this, you don't have that problem.
  - edgy 16y ago
    There is a big difference between old style 3D (that games have had for dozen years) and stereoscopic 3D where your brain captures the depth because each eye sees a _different_ image. Seeing depth makes it a much more involving experience.
    - arrogant 16y ago
      I agree, there is. Stereoscopic 3D isn't an end-all solution, however, and this isn't just ordinary "old style 3D", either. Head-tracking 3D is still a pretty novel technique, adding the illusion that you can look around an environment just by moving. I imagine that combining the two you could create a pretty immersive experience, although I can also admit there might be some difficulties doing so.
    - toast76 16y ago
      Do you still see 3D with one eye closed? Most people instinctively say "no". But the reality is that there are far more cues for depth than just stereoscopic vision. Someone with one eye still perceives depths by judging relative object sizes, changes in focus and subtle changes in objects when we move relative to the object. Head tracking will almost certainly play a big part in coming 3D techs (with or without stereoscopic tech). Add to this eye tracking to adjust virtual focus, and you start to get "real" 3D. One day!
edgy 16y ago
dont be fooled, this is not 3D as in "stereoscopic" 3D like avatar. You will not get the feeling of depth because for that you need a different image for each eye (only achieved with glasses or a special screen (like 3DS or at the movies). This is just a 3D geometry trick...
- jodrellblank 16y ago
  Do be fooled. What do you gain by refusing to be fooled?
  - edgy 16y ago
    Well I'm just annoyed because many people still say that the current 3D TVs dont bring anything, or that the technology with glasses will disappear soon. They think that someone will find a magic software trick to make it 3D without glasses, maybe by tracking eyes. But it is impossible, it is a misunderstanding on how stereoscopic 3D actually works. Current 3D technology is actually really good, though the glasses will surely get much lighter.
    - function_seven 16y ago
      You're probably correct in saying that glasses won't disappear anytime soon, and it is impossible with eye/head tracking alone to achieve stereoscopic 3D, but I'd like to bet that sometime in the future a holographic display could be created. One that sends different images to different regions of the room, such that your left eye is literally seeing a different view than your right eye, without shutters or polarization to mimic it. To truly get there, you'd have to actually change the path length of each ray, to allow your eyes' focus to agree with their convergence (a common complaint with current 3D, where the eyes are forced to focus on light coming from 3 meters away but converge on an object thats "1 meter" away.)
- newhouseb 16y ago
  While I undersand that 3D is a lot easier to say than "stereoscopic", I don't think any 3D movies warrant being called any more three dimensional than this sort of thing. IMHO, it's not 3D until I can move around and see different things (in each eye), i.e. until the medium understands itself in 3D space and reflects accordingly on my eyes.
jodrellblank 16y ago
Johnny Lee demonstrating Wii-based head tracking using the same plates-on-sticks imagery in 2007: http://www.youtube.com/watch?v=Jd3-eiid-Uw http://www.youtube.com/watch?v=Jd3-eiid-Uw Guy mounts two large framed LCDs in a wall and uses this effect to create a very effective virtual window which adjusts the view depending on viewer's position: http://www.engadget.com/2010/04/15/winscape-virtual-window-features-wiimote-headtracking-absolutel/ http://www.engadget.com/2010/04/15/winscape-virtual-window-f... ("Winscape") in April 2010.
dengzhi 16y ago
that is sick
javanix 16y ago
This is pretty cool. I'm most impressed that the iPad 2 has the horsepower to do all of the face-tracking and geometry rendering. As others have mentioned here, Johnny Lee had something similar running with the Wii hardware but as far as I remember the display was all driven by a PC which was (presumably) much more powerful than the iPad.
- arrogant 16y ago
  Johnny Lee's version was definitely cool, but it was more of a clever hack. It took advantage of the fact that the Wiimote is basically a very cheap IR camera, which normally tracks the IR light coming from the sensor bar. By using the Wiimote as a stationary camera, and wearing infrared LEDs on his face (attached to safety goggles, iirc) he could track the position of user's head. I'd imagine the Wii hardware would definitely be up to that task, as it is basically just the inverse of what it does already. Also, if you really want to be impressed with the iPad's rendering capabilities, check out Infinity Blade. That will blow your mind.
adriand 16y ago
This looks like it would support some fun new game mechanics on the iPad. I can imagine some sort of game where you have to look at the screen from different angles in order to perceive the route that your character (or a ball, vehicle, etc.) ought to take. Even something like 3D Pong would be kind of neat.
spyder 16y ago
So you have to constantly move your head to have some 3D illusion, otherwise if you not moving your head it's just plain 2D. It would be more interesting to combine with stereoscopic 3D, but even with this we are still far from real 3D. The next interesting thing will be real hologram displays using rewritable holograms: http://www.youtube.com/watch?v=b84RBl-jgZM http://www.youtube.com/watch?v=b84RBl-jgZM
- simonsquiff 16y ago
  Not just moving your head...you'll also get the effect from moving just the iPad. Will be interesting how much of the effect holds up from just subtle movement of either.
regularfry 16y ago
Does anyone have a reference for the camera transforms this is doing? I've got as far as figuring out that it's using an asymmetric frustum, but plugging the "obvious" numbers into OpenGL gave me obviously wrong results.
prr 16y ago
Mr Doob tried something similar a while back... http://mrdoob.com/blog/post/643 http://mrdoob.com/blog/post/643
romaniv 16y ago
As someone said, this was done for Wii several years ago. I think the idea is more practical for iPad, since you can rotate the screen instead of moving around. I would probably be more likely to use this if the position tracking was done via gyroscope.
neuroelectronic 16y ago
That hurts my retenal neurons just thinking about perceiving that.

275 points by jashmenn 16y ago

deleted 16y ago

[deleted]

jerf 16y ago

See also http://www.youtube.com/watch?v=h5QSclrIdlE http://www.youtube.com/watch?v=h5QSclrIdlE , which is actually a product that you can buy, though I think it was never localized for the US. Edit: Also, http://johnnylee.net/projects/wii/ http://johnnylee.net/projects/wii/ , under "Head Tracking for Desktop VR Displays using the Wii Remote" (down a bit). (Also, to be clear, I just mean these as interesting "see alsos". I am not saying "it's been done" (so what?) or "it doesn't work" (I have no direct experience to make a judgment). Nor am I saying anyone accused me otherwise. The iPad is still a potential interesting sweet spot for this tech.)

arrogant 16y ago
http://kotaku.com/#!5541044/looksleys-line-up-micro+review-broken-promise http://kotaku.com/#!5541044/looksleys-line-up-micro+review-b... Looks like it did come out in the US, although from the review it looks like we might not have heard about it because the effect just didn't work well enough. Maybe this implementation will deliver? It looks fairly promising, and the effect looks like it might be more pronounced on the (compared to the DSi) large screen of the iPad. Of course, I'd like to pass judgement on it personally, so here's hoping for a demo we can hold in our hands soon.
- kenjackson 16y ago
  This seems like a big deal even if it didn't work in the previous attempts. In part because I feel like it eventually will work. I'm excited and impressed by it. Although I do get a little disappointment that I didn't think of it. What the heck is my brain doing in my free time?!
  - jodrellblank 16y ago
    What the heck is my brain doing in my free time?! If you're bored... people ask for a touch sensitive rear surface - how about using the rear camera for gesture control? Wave your finger in front of it to scroll, that kind of thing. Also, some acoustic gesture recognition would be good to pick up one/two/three fingernail taps on the back.
    - kenjackson 16y ago
      Apple's already patented it. I don't want to get sued into oblivion. :-) http://www.addictivetips.com/mobile/iphone-control-through-camera-gestures-apple-patent-reveals/ http://www.addictivetips.com/mobile/iphone-control-through-c...
      - jodrellblank 16y ago
        That's annoying, but it only seems to cover tap detection by accelerometer where I was thinking of tap detection by audio click, which definitely seems plausible from a quick test of the sound recorder. Also it only covers finger swipe detection/direction determining, where I was also imagining gestures from holding fingers a few centimeters away, such as scissoring two fingers, thumb/finger flick, two finger scroll movement, two fingertips touching, etc. A quick test with the iPhone camera suggests that might have to be too carefully positioned to be practical, but should be possible at some level.
  - jurjenh 16y ago
    I'm excited and impressed by it mainly because I did think of it and started implementing it, thinking to develop it as a desktop interface, or possibly a cad program interface. Lots of linear algebra involved... However, I stopped mainly because at the time (about 3-4 years ago) my webcam and opencv introduced a delay that was unacceptable and caused visual jarring in a small application (I think I was using a window of 320x240 or so). I've also seen a demo done in flash by MrDoob of Three.js fame (IIRC), so it's been done by several people. I haven't really seen a useful implementation yet, so this is getting more interesting with these smaller, higher spec integrated devices...

deleted 16y ago

[deleted]

Dramatize 16y ago

It's starting to feel like the future.

rms 16y ago
Yup. I'm wondering when more average people are going to start noticing this.
- idonthack 16y ago
  I'm sure your father was saying the same thing forty years ago.
  - Dramatize 16y ago
    But this is what I was told the future would be when I was a kid.

MichaelApproved 16y ago

This could be extended with eye tracking to create even more impressive functions.

Devilboy 16y ago
I doubt the iPad's camera is good enough for eye tracking but maybe in future devices.

hook 16y ago

This is pretty amazing. I think that this coupled with multi-touch interactivity could be pretty revolutionary. It turns the iPad into an "iWindow".

edw 16y ago
Yes! As I was looking at the cube, I wanted the viewer to reach out and manipulate it: spin, pinch, what-have-you. I think the worst thing that could happen is that this sort of experience get baked into the core of the iPad's UI (or the UI of any tablet). But I could see imaging applications (MRIs?) where it would be valuable.
- rflrob 16y ago
  I suspect that touching the screen would, at all but one depth plane, break the illusion. The brain is eerily good at fooling itself, until it receives contradictory information.

evo_9 16y ago

Is this more or less what the new nintendo handheld does? How do they differ if not?

jarin 16y ago
The 3DS screen is closer to one of those lenticular "3d" things you'd see on a ruler or a bookmark as a kid. It uses two interlaced images that are fed to each eye separately. The iPad 2 screen is just a normal screen. http://loot-ninja.com/2010/06/22/how-the-nintendo-3ds-screen-works/ http://loot-ninja.com/2010/06/22/how-the-nintendo-3ds-screen... Basically, the iPad 2 demo only gives the illusion of being 3D when you move it, but since your head and hands are pretty much constantly moving it should look "mostly 3D" if it has enough sensitivity.
- evo_9 16y ago
  Thx, yeah I knew the ipad screen was a standard display, but having only seen ads for the new nintendo I wasn't sure what their tech was.
arrogant 16y ago
I'm pretty sure that the 3DS isn't doing any head tracking. If you move your head much at all the 3D effect breaks; you have to be looking practically dead at it.
- lazerwalker 16y ago
  Yep. The way the 3D effect works, there are essentially physical barriers inside the screen separating the different views being rendered for each eye. Move your head so that you're not viewing the screen straight-on and the barriers don't work properly, ruining the effect.

franck 16y ago

I wonder what happens when several people try and look over your shoulder. How does it handle multiple heads?

SammoJ 16y ago
It could only render for a single viewpoint. I would imagine the simplest approach (i.e. without tracking) would be to render for the viewpoint of the closest detected head.

drivebyacct2 16y ago

So this is "3d" now? Wow, I've got a lot of cool stuff that is 3d, including a prototype compiz plugin that does roughly the same thing. Gives you spatial context for windows on your desktop environment via Kinect and OpenCL. I had no idea I could get away with calling this "3d", and have people in the comments thinking it's the same technology as the 3ds.

arrogant 16y ago
3D on a screen is always an illusion of some sort. The 3DS is just a different illusion (albeit a more technically involved one, for sure). It's not any more "3D" than this. Perspective not changing when your head moves is one of the things that breaks a 3D illusion; with this, you don't have that problem.
- edgy 16y ago
  There is a big difference between old style 3D (that games have had for dozen years) and stereoscopic 3D where your brain captures the depth because each eye sees a _different_ image. Seeing depth makes it a much more involving experience.
  - arrogant 16y ago
    I agree, there is. Stereoscopic 3D isn't an end-all solution, however, and this isn't just ordinary "old style 3D", either. Head-tracking 3D is still a pretty novel technique, adding the illusion that you can look around an environment just by moving. I imagine that combining the two you could create a pretty immersive experience, although I can also admit there might be some difficulties doing so.
  - toast76 16y ago
    Do you still see 3D with one eye closed? Most people instinctively say "no". But the reality is that there are far more cues for depth than just stereoscopic vision. Someone with one eye still perceives depths by judging relative object sizes, changes in focus and subtle changes in objects when we move relative to the object. Head tracking will almost certainly play a big part in coming 3D techs (with or without stereoscopic tech). Add to this eye tracking to adjust virtual focus, and you start to get "real" 3D. One day!

edgy 16y ago

dont be fooled, this is not 3D as in "stereoscopic" 3D like avatar. You will not get the feeling of depth because for that you need a different image for each eye (only achieved with glasses or a special screen (like 3DS or at the movies). This is just a 3D geometry trick...

jodrellblank 16y ago
Do be fooled. What do you gain by refusing to be fooled?
- edgy 16y ago
  Well I'm just annoyed because many people still say that the current 3D TVs dont bring anything, or that the technology with glasses will disappear soon. They think that someone will find a magic software trick to make it 3D without glasses, maybe by tracking eyes. But it is impossible, it is a misunderstanding on how stereoscopic 3D actually works. Current 3D technology is actually really good, though the glasses will surely get much lighter.
  - function_seven 16y ago
    You're probably correct in saying that glasses won't disappear anytime soon, and it is impossible with eye/head tracking alone to achieve stereoscopic 3D, but I'd like to bet that sometime in the future a holographic display could be created. One that sends different images to different regions of the room, such that your left eye is literally seeing a different view than your right eye, without shutters or polarization to mimic it. To truly get there, you'd have to actually change the path length of each ray, to allow your eyes' focus to agree with their convergence (a common complaint with current 3D, where the eyes are forced to focus on light coming from 3 meters away but converge on an object thats "1 meter" away.)
newhouseb 16y ago
While I undersand that 3D is a lot easier to say than "stereoscopic", I don't think any 3D movies warrant being called any more three dimensional than this sort of thing. IMHO, it's not 3D until I can move around and see different things (in each eye), i.e. until the medium understands itself in 3D space and reflects accordingly on my eyes.

jodrellblank 16y ago

Johnny Lee demonstrating Wii-based head tracking using the same plates-on-sticks imagery in 2007: http://www.youtube.com/watch?v=Jd3-eiid-Uw http://www.youtube.com/watch?v=Jd3-eiid-Uw Guy mounts two large framed LCDs in a wall and uses this effect to create a very effective virtual window which adjusts the view depending on viewer's position: http://www.engadget.com/2010/04/15/winscape-virtual-window-features-wiimote-headtracking-absolutel/ http://www.engadget.com/2010/04/15/winscape-virtual-window-f... ("Winscape") in April 2010.

dengzhi 16y ago

that is sick

javanix 16y ago

This is pretty cool. I'm most impressed that the iPad 2 has the horsepower to do all of the face-tracking and geometry rendering. As others have mentioned here, Johnny Lee had something similar running with the Wii hardware but as far as I remember the display was all driven by a PC which was (presumably) much more powerful than the iPad.

arrogant 16y ago
Johnny Lee's version was definitely cool, but it was more of a clever hack. It took advantage of the fact that the Wiimote is basically a very cheap IR camera, which normally tracks the IR light coming from the sensor bar. By using the Wiimote as a stationary camera, and wearing infrared LEDs on his face (attached to safety goggles, iirc) he could track the position of user's head. I'd imagine the Wii hardware would definitely be up to that task, as it is basically just the inverse of what it does already. Also, if you really want to be impressed with the iPad's rendering capabilities, check out Infinity Blade. That will blow your mind.

adriand 16y ago

This looks like it would support some fun new game mechanics on the iPad. I can imagine some sort of game where you have to look at the screen from different angles in order to perceive the route that your character (or a ball, vehicle, etc.) ought to take. Even something like 3D Pong would be kind of neat.

spyder 16y ago

So you have to constantly move your head to have some 3D illusion, otherwise if you not moving your head it's just plain 2D. It would be more interesting to combine with stereoscopic 3D, but even with this we are still far from real 3D. The next interesting thing will be real hologram displays using rewritable holograms: http://www.youtube.com/watch?v=b84RBl-jgZM http://www.youtube.com/watch?v=b84RBl-jgZM

simonsquiff 16y ago
Not just moving your head...you'll also get the effect from moving just the iPad. Will be interesting how much of the effect holds up from just subtle movement of either.

regularfry 16y ago

Does anyone have a reference for the camera transforms this is doing? I've got as far as figuring out that it's using an asymmetric frustum, but plugging the "obvious" numbers into OpenGL gave me obviously wrong results.

prr 16y ago

Mr Doob tried something similar a while back... http://mrdoob.com/blog/post/643 http://mrdoob.com/blog/post/643

romaniv 16y ago

As someone said, this was done for Wii several years ago. I think the idea is more practical for iPad, since you can rotate the screen instead of moving around. I would probably be more likely to use this if the position tracking was done via gyroscope.

neuroelectronic 16y ago

That hurts my retenal neurons just thinking about perceiving that.