9 ms·
Glasses-free 3D using webcam head tracking
- il_nets 1y agoHi HN, I'm Sten, one of the creators. We built this because we wanted 3D experiences without needing a VR headset. The approach: use your webcam to track where you're looking, and adjust the 3D perspective to match. Demo: https://portality.io/dragoncourtyard/ https://portality.io/dragoncourtyard/ (Allow camera, move your head left/right) It creates motion parallax - the same depth cue your brain uses when you look around a real object. Feels like looking through a window instead of at a flat screen. Known limitations: - Only works for one viewer at a time - Needs decent lighting - Currently WebGL only We're still figuring out where this is genuinely useful vs just a novelty. Gaming seems promising, also exploring education and product visualization. Happy to answer questions!
- jampekka 11mo agoThe demo gives just a blank screen on Android Firefox, Kiwi and Chrome.
- kgarten 11mo agoIt works for me, just needs a time to load.
- aname2 11mo agohow long on what internet connection? i m on 1min and counting on 50mbit. but maybe it doesnt work on ubuntu 24 + firefox? should be webgl capable though. ok took around 2min for me to load, then works.
- Tade0 11mo agoDevtools say 40.23 MB / 13 MB transferred. Even on 1000Mbps it needed a moment. Works for me on Ubuntu 24 and Firefox.
- jampekka 11mo agoAh, thanks! Got it now on a better connection. A loading indicator would ease confusion.
- il_nets 11mo agoIt seems to vary a lot depending on device. It's just a "basic demo" for now, but yes good idea, thanks!
- socalgal2 11mo agoI don't know if you designed it for a specific monitor but, feedback. It tried using it on my M1 Mac. First thing, there is no loading indicator and it takes too long to start so I though it was broken a few times before I realized I had to wait longer. Second thing, although it was clearly tracking my head and moving the camera it did not make me feel like I'm looking into a window into a 3d scene behind the monitor. These kinds of demos have been around before. I don't know why some work better than others. some others: https://discourse.threejs.org/t/parallax-effect-using-face-tracking/16625 https://discourse.threejs.org/t/parallax-effect-using-face-t... https://www.anxious-bored.com/blog/2018/2/25/theparallaxview-illusion-of-depth-by-3d-head-tracking-on-iphone-x https://www.anxious-bored.com/blog/2018/2/25/theparallaxview...
- svantana 11mo agoIsn't it because the webcam FOV is unknown, which is needed to estimate distance from pixel (along with face size, but that should be less variant). The three.js demo had the strength parameter that can be used to calibrate. The iphone app is pre-calibrated for the most common devices, I believe.
- il_nets 11mo agoThanks for sharing all this feedback! And we will check out these links, appreciate you sharing it!
- president_zippy 11mo agoIt was definitely useful and appreciated on the "New" Nintendo 3DS XL, which also used a camera to track your eye movements and adjust the divergence accordingly. I hate the fact that Nintendo abandoned this technology because experiencing Ocarina of Time and Star Fox 64 in 3D was world-changing to me. I'd say I'm not the only one who misses this technology in games, because a used New 3DS XL costs at least $200 on eBay right now, which is more than what I paid new. I always thought 3D would combine really nicely with a ray traced graphics full of bright colors and reflections, similar to all those ray tracing demos with dozens of glossy marbles.
- Auracle 11mo agoSamsung recently released a monitor with similar technology, I believe, as a FYI.
- nxobject 11mo agoI'm surprised there's still a market for non-VR consumer 3D! I remember the post-Avatar rush of 3D-related products that never quite panned out.
- kevindamm 11mo agoI remember the 3D glasses that you could plug into the Sega Master System in the mid-80s. They took what would be interlaced frames and rendered them to different eyes instead (which made the version getting shown on the connected TV pretty trippy too). And then there was the time travel arcade game (also by Sega) that used a kind of Pepper's Ghost effect to give the appearance of 3D without glasses. That was in the early 90s. I think the idea of 3D displays keeps resurfacing because there's always a chance that the tech has caught up to people's dreams, and VR displays sure have brought the latency down a lot but even the lightest headsets are still pretty uncomfortable after extended use. Maybe in another few generations... but it will still feel limiting until we have holodeck-style environments IMO.
- 11mo ago
- LordHeini 11mo agoThe obvious use case would be to replace the clunky head tracking systems which are often used in simulator games. Systems like trackir, which require dedicated hardware.
- dTal 11mo agoYou can do this today with OpenTrack: https://github.com/opentrack/opentrack https://github.com/opentrack/opentrack Also, TrackIR is just an IR webcam, IR leds, and a hat with reflectors. You can DIY the exact same setup easily with OpenTrack, but OpenTrack also has a neural net webcam-only tracker which is, AFAIK, pretty much state of the art. At any rate it works incredibly robustly. Actually I have already used it to implement the same idea as the post, with the added feature of anaglyph (red/blue) glasses 3D. The way I did it, I put an entire lightfield into a texture and rendered it with a shader. Then I just piped the output of OpenTrack directly into the shader and Robert, c'est votre proverbial oncle. The latency isn't quite up to VR standard (the old term for this is "fishtank VR"), but it's still quite convincing if you don't move your head too fast.
- jojobas 11mo agoTrackir is just a camera with an infrared led.
- trenchpilgrim 11mo agoThere's already a wide variety of Opentrack plugins that use everything from off the shelf webcams to DIY infrared trackers to an iPhone app and FaceID/AirPods.
- Tade0 11mo agoI can confirm that is works decently well with a sunny roof window in the background, which is normally enough for people to complain that my face is too dark. 8yo me, who instinctively tried to look behind the display's field of view during intense gaming sessions, would appreciate this feature very much. My belief is that if it shifted the pov to a lesser degree than in the demo, people generally wouldn't notice, but still subconsciously register this as a more immersive experience. I'm also glad that the web version doesn't try to cook my laptop - good work.
- il_nets 11mo agoThanks! If you click "Menu" and then "Settings" you can play around with e.g. the sensitivity. Ideally we'd automatically optimize the calibration according to for example what device you are using, but that's something we would do a bit more long-term. Appreciate it!
- utopcell 11mo agodefinitely laggy, but works even under low lighting conditions and a camera that is not facing straight forward.
- il_nets 11mo agoMay I ask what device and browser you are using?
- toss1 11mo agoVery cool! I can see this quite useful for educational demonstrations of physics situations and mechanical systems (complex gearing, etc.). Also maybe for product simulations/demonstrations in the design phase — take output from CAD files and make a nice little 3D demo. Maybe have an "inertia(?)" setting that makes it keep turning when you move far enough off center, as if you were continuing to walk around it. The single-viewer limitation seems obvious and fundamental, and maybe a bit problematic for the above use cases, such as when showing something to a small group of people. One key may be to take steps to ensure it robustly locks onto and follows only one face/set of eyes. It would be jarring to have it locking onto different faces as conditions or positions subtly change.
- il_nets 11mo agoGood ideas - we've considered these as well actually! The intertia-idea wouldn't be too difficult to implement, but its usefulness would probably depend on the application area. Yep exactly. Usually it locks onto one person's face but it can also jump around, so there are still optimizations we can do there - but generally it's supposed to be for one person. If you compare to VR headsets, two people can't wear the same VR headset anyway!
- nine_k 11mo agoThis is fun! But I see it showing me a close-up view (smaller FOV) if I move my head back, and a wider view (larger FOV) if I bring my head closer. This is the opposite to what I'd expect: if I bring my head nearer to the screen, I should see a more detail, closer (narrower FOV).
- Wowfunhappy 11mo agoYour expectation doesn't match real life! Try it with a window. If you bring your head closer to the window, your FOV increases, ie you can see more of the scene outside the window.
- nine_k 11mo agoBut there is no window. There is a box with a dragon in it. When I bring my eyes closer to the box, I expect to see more of the box, and more of the dragon. You can try it with any real box; substitute any appropriate object for the dragon. Pick your favorite 3D environment that features walls and dragons; notice how both grow, covering more of your FOV. This is what I expect to see as I draw closer to the screen.
- marcodiego 11mo agoHow about contributing that to Godot?
- il_nets 11mo agoDefinitely! Our current focus is on Unity, because that's what we're most used to, but we'd also build the solution for at least Unreal and Godot as well!
- zipy124 11mo agoThe latency is unfortunately bad enough that it prevents me from getting any depth illusion :( I've seen other implementations where it works really well though.
- il_nets 11mo agoMay I ask what device you are trying it on? In our experience, it can vary depending on what device - something we're looking to improve on further down the line. Do you remember which other implementations you've seen that worked really well?
- arnarbi 11mo agoBack in college (~2008) we implemented this with a 7 foot tall back-projected screen and a couple of Wii remotes after seeing Johnny Lee’s video. The nice thing with that screen was that you could stand so close to it you couldn’t really see the edges. We had as many people come test as we could, and we found that 90% of them didn’t get a sense of depth, likely because it lacked stereo-vision cues. It only worked for folks with some form of monocular vision, incl myself, who were used to relying primarily on other cues like parallax.
- aname2 11mo agoreally fancy idea and cool project! Its a bit stuttery probably due to the 30fps of a webcam but works. Feels slightly weird somehow though, probably from the lag. update: just tried to open the site again now and its gone but leads to some kind of shop? update2: oh use the link in the comment for the demo: https://portality.io/dragoncourtyard/ https://portality.io/dragoncourtyard/
- brenschluss 11mo agoBTW, the OG demo from Johnny Lee is from 2007: https://www.youtube.com/watch?v=Jd3-eiid-Uw https://www.youtube.com/watch?v=Jd3-eiid-Uw
- poglet 11mo agoThanks for posting, I was sure I recalled something like this form a long time ago. I also build my self a FreeTrack headset (https://en.wikipedia.org/wiki/FreeTrack https://en.wikipedia.org/wiki/FreeTrack) around this same time to play the Arma / Operating Flashpoint games using IR LED's attached to a hat that my webcam would track.
- il_nets 11mo agoYou're probably recalling OG Johnny Lee video: https://www.youtube.com/watch?v=Jd3-eiid-Uw https://www.youtube.com/watch?v=Jd3-eiid-Uw That's cool, what were the benefits/drawbacks of that compared to other VR headsets would you say?
- xandman 11mo agoYeah, this came out during the last few weeks of my time in high school. It is a major reason I got into computer science and became a programmer. Good Times!
- il_nets 11mo agoWow that's really cool!
- duckerduck 11mo agoWow, thanks for sharing, that really takes me back. I was so hyped when I saw this as a kid, my dad and I made a mount for my glasses with two IR LEDs and a battery. I remember that I was super impressed with the effect.
- AndrewOMartin 11mo ago
- epaga 11mo agoInterestingly, for this parallax 3D effect to work, the head tracking needs to basically move "backwards" from typical head tracking since it needs to keep the focal point the same, if I'm understanding correctly. Any time I've tried this out it's fun and would likely be most useful for something like a 3D painting you hang on your wall. Related idea, but not the same, might be my iOS and Android app that uses your phone's AR for head tracking and then sends that data to your PC for smooth sim game head tracking. https://smoothtrack.app https://smoothtrack.app
- mrheosuper 11mo agoIs this similar to sony spatial 3d display ?
- pezezin 11mo agoMy company bought one of those Sony Spatial Reality Displays a couple of weeks ago and I have to say it is truly impressive, the 3D effect is really convincing. I just tried this demo and it is cool, but nowhere as good.
- il_nets 11mo agoWe will have to check this out - could you share what the main differences are / in what ways Sony Spatial Reality Displays are better?
- pezezin 11mo agoThe Sony display uses a parallax barrier with eye tracking, like a Nintendo 3DS on steroids. It projects a different image for each eye, there is no way you can replicate the effect with a standard display unless you use shutter glasses.
- rspoerri 11mo agoFrom my experience this only ever looks good in a recorded video. Because we are used to assume that the real object captured within the video is perceivable in a stereoscopic form. But it‘s not. The real usage of head and viewtracking is as a controller, not to improve immersion.
- rullelito 11mo agoDrawback is that it only works if you constantly move your head or device.
- il_nets 11mo agoHow do you mean? Your perspective when looking out of a window also only changes when you move your head...
- quitit 11mo agoReminds me of the Amazon Fire phone which featured similar tech prominently. "Dynamic perspective". https://www.youtube.com/watch?v=6trOg2IK2Zg https://www.youtube.com/watch?v=6trOg2IK2Zg
- hex4def6 11mo agoTangentially involved in that project. One of the big issues with that phone was that in order to do dynamic perspective, you're having to run a 3D render at 60fps constantly. That's a huge power hog, and prevents you from doing many of the power savings techniques you otherwise could on a normal phone -- shutting down the GPU, reduced refresh rate, heck, even RAM backed displays.
- richrichardsson 11mo ago> This content is hosted by a third party provider that does not allow video views without acceptance of Targeting Cookies. Please set your cookie preferences for Targeting Cookies to yes if you wish to view videos from these providers. OK, that's a no then.
- boredpudding 11mo agoIt's YouTube. This is super common, just worded in a weird way.
- kimsey0 11mo agoIt's also incorrect. They could just use youtube-nocookie.com instead if tracking cookies are disabled. https://support.google.com/youtube/answer/171780?hl=en#zippy=%2Cturn-on-privacy-enhanced-mode https://support.google.com/youtube/answer/171780?hl=en#zippy...
- bangaladore 11mo agoThe funny thing is I'm unable to navigate the UI to see the video. I disabled (enabled) targeted cookies, and it still doesn't show.
- thenthenthen 11mo agoYeah what is up? When i accept cookies i find myself at the bottom of the page at a checkout button with a dollar price number? This confused me, after three more tries I gave up.
- globalnode 11mo agoThats the first time I've seen that, its a no for me too then. Good to know I'm being targeted though :)
- il_nets 11mo agoHonestly I have no idea what the problem seems to be - we're not collecting any cookies. Also, there's no video, so I'm not sure what you are referring to... Would you be able to clarify?
- stavros 11mo agoDoes this really fool you if you have two eyes? I haven't been able to try it or watch the videos (they're behind a cookie warning I can't get past).
- spwa4 11mo agoIf the latency is low enough to fool you, it makes the screen look like a window. It's much more limited than a headset, but especially with a natural scene, it does get the illusion across. But you're still behind the window.
- papes_ 11mo agoI remember this being done for a DSIware game released ~2010 - I couldn't find much footage apart from this quick clip from a trailer. https://youtu.be/4zZfsyHEcZA?si=BE2I991zEVxPEt9F&t=57 https://youtu.be/4zZfsyHEcZA?si=BE2I991zEVxPEt9F&t=57 Seemed to have some different names depending on region (Looksley's Line Up, Tales in a Box: Hidden Shapes in Perspective). I recall it working very well at the time.
- guyromm 11mo agohere's a webgl poc that was tested with mobile & desktop browsers: https://www.webgma.co.il/Articles/window-3d-tracking/en/ https://www.webgma.co.il/Articles/window-3d-tracking/en/ source code at https://github.com/guyromm/Window https://github.com/guyromm/Window imho this should be useful for driving/flight sims, giving the player the ability to lean inside the vehicle, changing their viewpoint on the surroundings.
- nunodonato 11mo agoHow is this different from the many available head tracking devices? I used to play Flight Simulator with head tracking[1]. It was great, although it did consume a bit of CPU cycles. IR trackers are more efficient [1] - https://www.youtube.com/watch?v=P07nIcczles https://www.youtube.com/watch?v=P07nIcczles (actually, this one was using a paper tracker because the face tracker had a big impact on fps)
- Mountain_Skies 11mo agoSeem to recall one of the pinball emulators having this as a plug-in years ago.
- il_nets 11mo agoOh cool! Can you remember what it was called?
- SirMaster 11mo agoBut does it actually make stuff look like you can reach out and touch it like the 3D movies at theme parks? I don't see how it would. This just seems like parallax. You can do this with a Kinect for head tracking and people do that with homemade pinball machines using a TV as the table, and a Kinect to track your head, so it looks like the table is 3D down into the TV. But I don't think it creates the full 3D effect with things looking like they are coming way out of the screen and like a tangible thing you can reach out and touch.
- deleted 11mo ago[deleted]
- Ninjinka 11mo agoCalling this 3D is a stretch, it does not make things appear like they're coming out of the screen. That requires having a way to show something different to each eye, something not possible with a standard display. I did see a demo of 3D without glasses on a full monitor that DID make it look like it was coming out of the screen at CES, it requires a $3000 monitor though: https://www.3dgamemarket.net/content/32-4k-glasses-free-3d-gaming-display https://www.3dgamemarket.net/content/32-4k-glasses-free-3d-g... Also the 3DS obviously.
- regularfry 11mo agoNot strictly true. Out past a certain distance your brain uses parallax for depth cues because the difference between each of the eye's images is too small. It's 3d, just not stereoscopic 3d.
- layla5alive 11mo agoShutter glasses would like to disagree about your assertion
- fpia 11mo agohead/eye tracking?
- il_nets 11mo agoYep, both. But using the eye's position to adjust the perspective.
- peterept 11mo agoI got excited about this a few years ago when I was into digital pinball. I built an open source project called Phonemote using an iPhone to track eyes and relay telemetry over WiFi/BT and a plugin for FuturePinball. However the result wasn’t that useful because we humans have 2 eyes. The 3d effect is very compelling if you close/cover one eye. But that becomes annoying quickly. The best results I had was sneering a little hair gel in the center of my left eye glassses. Then it felt like I was using 2 eyes but really my right eye was seeing the pinball table clearly and fooling my brain.
- smashah 11mo agoThere is a taiwanese company that has developed a sandwiched lenticular screen protector for phones that should fix this. Looks amazing in my opinion. You should try it out with your project.
- peterept 11mo agoOh yes at the time I ordered a sheet but it didn’t align with my display panel. I should try that again.
- anfractuosity 11mo agoDo you have a link to that, sounds cool?
- smashah 11mo agothis is the one i saw. https://www.optiqb.com/ https://www.optiqb.com/
- nicman23 11mo agothere was a compiz 0.8 plugin for that
- il_nets 11mo agoOh, do you have a link?
- nicman23 11mo agohttps://www.youtube.com/watch?v=kTNG1GN4VV8 https://www.youtube.com/watch?v=kTNG1GN4VV8 but cannot seem to find the actual repo anywhere
- nicman23 11mo agoafter some archaeology i found a mirror https://github.com/ethus3h/compiz-plugins-community/tree/master/src/headtracking https://github.com/ethus3h/compiz-plugins-community/tree/mas...