4 ms·

Show HN: Seamless head tracking for games using the TrueDepth camera (iOS)

29 points by epaga 6y ago

epaga 6y ago
This has been a fun project - moving the camera around in games like Flight Simulator simply by slightly moving your head is really immersive. Not quite VR, but good enough! Happy to answer any questions. (Coming soon - a version that will run without a TrueDepth camera.)
- ChicagoBoy11 6y ago
  I love projects like this. It's such an elegant and in hindsight immediately obvious solution that it definitely stops you in your tracks. Will try it this weekend with FS2020; certainly a good stop-gap until they implement native VR. Can you perhaps elaborate a bit on what the process was like getting some of that info through the iOS APIs and then exposing it through UDP for the PC? Any interesting snags you hit along the way?
  - epaga 6y ago
    Thanks! For me, and this surprised me a bit, the most frustrating parts actually weren’t any of the APIs but just plain old network headaches - it took me a bit to figure out the UDP protocol OpenTrack needs, but then it was often struggling with it not connecting or working only to 10 minutes later realize I had switched my phone‘s WiFi off while testing.
- PostPlummer 6y ago
  Bought it to support you. Hope it helps and thanks for the heads up (pun intended).
nxpnsv 6y ago
Looks awesome. I guess no mac games supports FreeTrack?
chrischen 6y ago
Is there any way to get this to work with spatial audio instead? I bought a $99 device that adds head tracking aiming to simulate surround sound audio by tracking my head movement, but they no longer support their software and it's super buggy!
jasonpeacock 6y ago
I'm sure I'm missing something as I have no experience with these tools/software.... When you turn your head to the right, the screen in front of you shows the view to the right but now you're facing to the right and you can't see the screen in front of you anymore? How do you setup your battlestation to best use this?
- evan_ 6y ago
  I was wondering this as well. In the video he turns his head about 20° or 30° and the view shifts almost 90°, so it must amplify his motions? I suppose you would get used to that pretty quickly.
  - rspicer 6y ago
    Yes. OpenTrack has user-editable Bézier curves that map real-world translation and rotation to what’s output to the game. This is an awesome hack and I wish I’d thought of it.
    - rco8786 6y ago
      OHHHH that's brilliant
- deleted 6y ago
  [deleted]
mikhuang 6y ago
This looks a lot more comfortable than wearing a QR code on my head. Is there expected to be a lot of accuracy difference between a TrueDepth cam and a regular cam?
Animats 6y ago
Nintendo 3DS doing that, 2010: [1] The "big window" form of this (2012) is especially cool.[2] [1] https://www.youtube.com/watch?v=dUC5JJCxx5k https://www.youtube.com/watch?v=dUC5JJCxx5k [2] https://youtu.be/V2hxaijuZ6w https://youtu.be/V2hxaijuZ6w
- mcphage 6y ago
  It never worked very well on the DSi—which is a shame, since I really liked the game that used it (Looksley's Line Up). The DSi camera is really low resolution (0.3 Megapixel). I've never seen it ported anywhere else, and I don't know if any emulators will run it. (It will run on the 3DS, but the camera is in a different place, and the camera placement is pretty essential to how the game works).
locusm 6y ago
Given the TrackIR product is more than $400 here in Australia this looks like a great alternative.
- Rebelgecko 6y ago
  If you're feeling brave, you can either build your own TrackIR (basically super glue a few LEDs to a hat). IIRC you can also use OpenTrack with a regular USB webcam
  - lostmsu 6y ago
    There was a project from Google just a few weeks ago, that is able to figure out position of irises with good precision using a web cam: https://www.reddit.com/r/MachineLearning/comments/i5mbdm/r_eyes_on_me_google_ai_mediapipe_iris_improves/ https://www.reddit.com/r/MachineLearning/comments/i5mbdm/r_e...
theNJR 6y ago
Whoa. I was about to spend the cash on hardware to support this in MSFS2020. Can’t wait to try.
epaga 6y ago
Update: I've got a version working on any iOS 13 device - here's a free public beta link if anyone would like to join https://testflight.apple.com/join/ytc1tAdA https://testflight.apple.com/join/ytc1tAdA

29 points by epaga 6y ago

This has been a fun project - moving the camera around in games like Flight Simulator simply by slightly moving your head is really immersive. Not quite VR, but good enough! Happy to answer any questions. (Coming soon - a version that will run without a TrueDepth camera.)

ChicagoBoy11 6y ago
I love projects like this. It's such an elegant and in hindsight immediately obvious solution that it definitely stops you in your tracks. Will try it this weekend with FS2020; certainly a good stop-gap until they implement native VR. Can you perhaps elaborate a bit on what the process was like getting some of that info through the iOS APIs and then exposing it through UDP for the PC? Any interesting snags you hit along the way?
- epaga 6y ago
  Thanks! For me, and this surprised me a bit, the most frustrating parts actually weren’t any of the APIs but just plain old network headaches - it took me a bit to figure out the UDP protocol OpenTrack needs, but then it was often struggling with it not connecting or working only to 10 minutes later realize I had switched my phone‘s WiFi off while testing.
PostPlummer 6y ago
Bought it to support you. Hope it helps and thanks for the heads up (pun intended).

nxpnsv 6y ago

Looks awesome. I guess no mac games supports FreeTrack?

chrischen 6y ago

Is there any way to get this to work with spatial audio instead? I bought a $99 device that adds head tracking aiming to simulate surround sound audio by tracking my head movement, but they no longer support their software and it's super buggy!

jasonpeacock 6y ago

I'm sure I'm missing something as I have no experience with these tools/software.... When you turn your head to the right, the screen in front of you shows the view to the right but now you're facing to the right and you can't see the screen in front of you anymore? How do you setup your battlestation to best use this?

evan_ 6y ago
I was wondering this as well. In the video he turns his head about 20° or 30° and the view shifts almost 90°, so it must amplify his motions? I suppose you would get used to that pretty quickly.
- rspicer 6y ago
  Yes. OpenTrack has user-editable Bézier curves that map real-world translation and rotation to what’s output to the game. This is an awesome hack and I wish I’d thought of it.
  - rco8786 6y ago
    OHHHH that's brilliant
deleted 6y ago
[deleted]

mikhuang 6y ago

This looks a lot more comfortable than wearing a QR code on my head. Is there expected to be a lot of accuracy difference between a TrueDepth cam and a regular cam?

Animats 6y ago

Nintendo 3DS doing that, 2010: [1] The "big window" form of this (2012) is especially cool.[2] [1] https://www.youtube.com/watch?v=dUC5JJCxx5k https://www.youtube.com/watch?v=dUC5JJCxx5k [2] https://youtu.be/V2hxaijuZ6w https://youtu.be/V2hxaijuZ6w

mcphage 6y ago
It never worked very well on the DSi—which is a shame, since I really liked the game that used it (Looksley's Line Up). The DSi camera is really low resolution (0.3 Megapixel). I've never seen it ported anywhere else, and I don't know if any emulators will run it. (It will run on the 3DS, but the camera is in a different place, and the camera placement is pretty essential to how the game works).

locusm 6y ago

Given the TrackIR product is more than $400 here in Australia this looks like a great alternative.

Rebelgecko 6y ago
If you're feeling brave, you can either build your own TrackIR (basically super glue a few LEDs to a hat). IIRC you can also use OpenTrack with a regular USB webcam
- lostmsu 6y ago
  There was a project from Google just a few weeks ago, that is able to figure out position of irises with good precision using a web cam: https://www.reddit.com/r/MachineLearning/comments/i5mbdm/r_eyes_on_me_google_ai_mediapipe_iris_improves/ https://www.reddit.com/r/MachineLearning/comments/i5mbdm/r_e...

theNJR 6y ago

Whoa. I was about to spend the cash on hardware to support this in MSFS2020. Can’t wait to try.

epaga 6y ago

Update: I've got a version working on any iOS 13 device - here's a free public beta link if anyone would like to join https://testflight.apple.com/join/ytc1tAdA https://testflight.apple.com/join/ytc1tAdA