6 ms·

Hand-Tracking with Three.js

135 points by marban 4y ago

butlersean 4y ago
the hand tracking is spot on by the small image in the bottom left hand corner. steady and accurate positioning of two hands. :-) unfortunately the rendering of the hands in the large window jumps all over the place on firefox, ubuntu, razer laptop. :-(
- rikroots 4y ago
  It seems quite stable to me (on Chrome) ... until I weave my fingers from each hand together. That said, I still think MediaPipe is an excellent piece of ML tech. And, from a dev point of view, quite easy to get working for various things in the browser[1][2] [1] - MediaPipe Selfie Segmentation for real-life background replacement - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-001.html https://scrawl-v8.rikweb.org.uk/demo/mediapipe-001.html [2] - MediaPipe Face Mesh for, well, drawing lines on your face - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-003.html https://scrawl-v8.rikweb.org.uk/demo/mediapipe-003.html
- icoder 4y ago
  Same problem on Chrome (mac M1). For weird positions I can understand as the computer must still interpret the 2D dots in the small screen to 3D coordinates, but even when I just show my hand fingers spread, palm facing the camera, the small image is stable and the 3D image glitches quite profoundly.
- ncr100 4y ago
  On my Pixel 7 Pro phone, I'm able to ge represent all kinds of strange hand positions correctly on the screen via my selfie camera. There is some jumpiness but I am holding the phone in one hand and making hand positions with my other so I'm less concerned about the jiggles. It has a hard time naturally when all of my fingers are occluded by one another. Way to go.
- dTal 4y ago
  Same problem. The hand tracking is really impressive, basically flawless. The rendered hand is all over the place, doesn't match the debug window at all.
speps 4y ago
It doesn't allow me to select which camera to use unfortunately, Chrome is set to use one but the site probably uses the first one it finds.
- radiojasper 4y ago
  You need a better browser. Firefox lets you choose which camera you want to use. https://jasper.monster/sharex/firefox_8DYyGO7U9f.png https://jasper.monster/sharex/firefox_8DYyGO7U9f.png
bhouston 4y ago
This is unfortunately a bit buggy. I can see in the 2D image that it is tracking when my hand is turned backwards and relatively flat to the camera. But the 3D is showing my hands are curled.
philipphutterer 4y ago
This is really cool to see. I don't have any problems or bugs using it, but it seems like the 3d rendering of the virtual hands is not quite as fast as the tracking itself. I wonder if and how this could be used in a meaningful way. As a feature for things like Google Quick Draw it would be fun.
klaussilveira 4y ago
Doesn't really work well while holding objects or faster movements, which I imagine would be restrictive for gaming or simulation purposes. Might be useful as a replacement for Leap Motion, though. I can see this working for manipulating a desktop environment.
- calny 4y ago
  Congrats OP, very cool and runs great for me in Chrome and Firefox. Mediapipe can indeed work for manipulating desktop environments, I've been working off and on at that for a couple years.[0] It's tricky to make the interactions effective and minimize false positives while also avoiding a heavy cognitive load on the user, but there's lots of potential. [0] https://www.youtube.com/watch?v=bHjj46AIVxs https://www.youtube.com/watch?v=bHjj46AIVxs
jackbach 4y ago
Hello! I am the creator of this experiment. Glad to see the conversation about hand tracking in the browser over here. This demos is done under the context of a series of creative experiments on how to use real time hand tracking in the browser for creative interactions. Will be posting more experiments soon. Tech background: I am using MediaPipe to control the hand rig in threejs. MediaPipe provides landmarks that are used to control a threejs Skeleton (hierarchy of bones with rotations). Feel free to ask, I will answer any questions!
- jimmySixDOF 4y ago
  This is amazing I had a browser plugin called flutter years back that was able to do webcam gesture recognition for scroll and forward/back. This is using threejs so I wonder how much is CPU or GPU and also how well this could, now or in the future, run under the hood in the background of a web game (or WebXR!) just as the input device and without too much overhead. Great Proof of Concept !!
  - jackbach 4y ago
    Thanks for the nice words! Your plugin sounds like fun. In terms of using hand tracking for web games: my next experiments will use this setup to interact with 3D scenes.
    - jimmySixDOF 4y ago
      Looking forward to where you take this, and just to clarify I was just an end user of Flutter but was checking for a link (I had this installed years back) and turns out they got aquired by Google[1] so could be somewhere in Lense for all I know! [1] https://www.searchenginewatch.com/2013/10/03/google-acquires-flutter-creator-of-hand-gesture-recognition-technology/ https://www.searchenginewatch.com/2013/10/03/google-acquires...
- marban 4y ago
  Based on the sample at https://google.github.io/mediapipe/solutions/hands https://google.github.io/mediapipe/solutions/hands — that doesn't even sound all too complex.
kinard 4y ago
Jack, it's amazing. Very impressive stuff.
xrd 4y ago
This is so awesome. Fantastic demo. An "air-piano" seems well within the realms of possibility now.
micheljansen 4y ago
Wow this works much better than I had expected. Well done!
xnx 4y ago
Cool mashup! For anyone interested, I found this codepen from Google where you can play with Mediapipe in your browser: https://codepen.io/mediapipe/pen/RwGWYJw https://codepen.io/mediapipe/pen/RwGWYJw
julienreszka 4y ago
fails when both hands touch or cross
futhey 4y ago
Cool progress, really nice work! Not quite high enough fidelity to handle ASL though. Some issues I ran into testing it, if it's something that interests you: - Cannot distinguish closed vs. open fingers (always adds gaps between fingers, even if they're touching) (B) - Can't handle crossed fingers (R) - Doesn't seem to like extended vs. curled fingers in some cases (H) - Other failed letters: (Q), (E?), (F?), (G), (Q), (S), (U/V) But, when signing naturally, it seems to get enough of the shapes and orientation correct enough to understand what I'm seeing. I'm sure there's things it'd trip up on because of some of the above weaknesses in detecting hand shapes but it does seem to get movement, orientation, and position "good enough".
stuntkite 4y ago
Really well done! I've taken a stab at this problem with mixed results. This is leaps and bounds beyond most of my attempts. Thanks for sharing!
Aspos 4y ago
I guess one could make a fun therminvox game out of this.

135 points by marban 4y ago

the hand tracking is spot on by the small image in the bottom left hand corner. steady and accurate positioning of two hands. :-) unfortunately the rendering of the hands in the large window jumps all over the place on firefox, ubuntu, razer laptop. :-(

rikroots 4y ago
It seems quite stable to me (on Chrome) ... until I weave my fingers from each hand together. That said, I still think MediaPipe is an excellent piece of ML tech. And, from a dev point of view, quite easy to get working for various things in the browser[1][2] [1] - MediaPipe Selfie Segmentation for real-life background replacement - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-001.html https://scrawl-v8.rikweb.org.uk/demo/mediapipe-001.html [2] - MediaPipe Face Mesh for, well, drawing lines on your face - https://scrawl-v8.rikweb.org.uk/demo/mediapipe-003.html https://scrawl-v8.rikweb.org.uk/demo/mediapipe-003.html
icoder 4y ago
Same problem on Chrome (mac M1). For weird positions I can understand as the computer must still interpret the 2D dots in the small screen to 3D coordinates, but even when I just show my hand fingers spread, palm facing the camera, the small image is stable and the 3D image glitches quite profoundly.
ncr100 4y ago
On my Pixel 7 Pro phone, I'm able to ge represent all kinds of strange hand positions correctly on the screen via my selfie camera. There is some jumpiness but I am holding the phone in one hand and making hand positions with my other so I'm less concerned about the jiggles. It has a hard time naturally when all of my fingers are occluded by one another. Way to go.
dTal 4y ago
Same problem. The hand tracking is really impressive, basically flawless. The rendered hand is all over the place, doesn't match the debug window at all.

speps 4y ago

It doesn't allow me to select which camera to use unfortunately, Chrome is set to use one but the site probably uses the first one it finds.

radiojasper 4y ago
You need a better browser. Firefox lets you choose which camera you want to use. https://jasper.monster/sharex/firefox_8DYyGO7U9f.png https://jasper.monster/sharex/firefox_8DYyGO7U9f.png

bhouston 4y ago

This is unfortunately a bit buggy. I can see in the 2D image that it is tracking when my hand is turned backwards and relatively flat to the camera. But the 3D is showing my hands are curled.

philipphutterer 4y ago

This is really cool to see. I don't have any problems or bugs using it, but it seems like the 3d rendering of the virtual hands is not quite as fast as the tracking itself. I wonder if and how this could be used in a meaningful way. As a feature for things like Google Quick Draw it would be fun.

klaussilveira 4y ago

Doesn't really work well while holding objects or faster movements, which I imagine would be restrictive for gaming or simulation purposes. Might be useful as a replacement for Leap Motion, though. I can see this working for manipulating a desktop environment.

calny 4y ago
Congrats OP, very cool and runs great for me in Chrome and Firefox. Mediapipe can indeed work for manipulating desktop environments, I've been working off and on at that for a couple years.[0] It's tricky to make the interactions effective and minimize false positives while also avoiding a heavy cognitive load on the user, but there's lots of potential. [0] https://www.youtube.com/watch?v=bHjj46AIVxs https://www.youtube.com/watch?v=bHjj46AIVxs

jackbach 4y ago

Hello! I am the creator of this experiment. Glad to see the conversation about hand tracking in the browser over here. This demos is done under the context of a series of creative experiments on how to use real time hand tracking in the browser for creative interactions. Will be posting more experiments soon. Tech background: I am using MediaPipe to control the hand rig in threejs. MediaPipe provides landmarks that are used to control a threejs Skeleton (hierarchy of bones with rotations). Feel free to ask, I will answer any questions!

jimmySixDOF 4y ago
This is amazing I had a browser plugin called flutter years back that was able to do webcam gesture recognition for scroll and forward/back. This is using threejs so I wonder how much is CPU or GPU and also how well this could, now or in the future, run under the hood in the background of a web game (or WebXR!) just as the input device and without too much overhead. Great Proof of Concept !!
- jackbach 4y ago
  Thanks for the nice words! Your plugin sounds like fun. In terms of using hand tracking for web games: my next experiments will use this setup to interact with 3D scenes.
  - jimmySixDOF 4y ago
    Looking forward to where you take this, and just to clarify I was just an end user of Flutter but was checking for a link (I had this installed years back) and turns out they got aquired by Google[1] so could be somewhere in Lense for all I know! [1] https://www.searchenginewatch.com/2013/10/03/google-acquires-flutter-creator-of-hand-gesture-recognition-technology/ https://www.searchenginewatch.com/2013/10/03/google-acquires...
marban 4y ago
Based on the sample at https://google.github.io/mediapipe/solutions/hands https://google.github.io/mediapipe/solutions/hands — that doesn't even sound all too complex.

kinard 4y ago

Jack, it's amazing. Very impressive stuff.

xrd 4y ago

This is so awesome. Fantastic demo. An "air-piano" seems well within the realms of possibility now.

micheljansen 4y ago

Wow this works much better than I had expected. Well done!

xnx 4y ago

Cool mashup! For anyone interested, I found this codepen from Google where you can play with Mediapipe in your browser: https://codepen.io/mediapipe/pen/RwGWYJw https://codepen.io/mediapipe/pen/RwGWYJw

julienreszka 4y ago

fails when both hands touch or cross

futhey 4y ago

Cool progress, really nice work! Not quite high enough fidelity to handle ASL though. Some issues I ran into testing it, if it's something that interests you: - Cannot distinguish closed vs. open fingers (always adds gaps between fingers, even if they're touching) (B) - Can't handle crossed fingers (R) - Doesn't seem to like extended vs. curled fingers in some cases (H) - Other failed letters: (Q), (E?), (F?), (G), (Q), (S), (U/V) But, when signing naturally, it seems to get enough of the shapes and orientation correct enough to understand what I'm seeing. I'm sure there's things it'd trip up on because of some of the above weaknesses in detecting hand shapes but it does seem to get movement, orientation, and position "good enough".

stuntkite 4y ago

Really well done! I've taken a stab at this problem with mixed results. This is leaps and bounds beyond most of my attempts. Thanks for sharing!

Aspos 4y ago

I guess one could make a fun therminvox game out of this.