4 ms·

Hello! I am the creator of this experiment. Glad to see the conversation about hand tracking in the browser over here. This demos is done under the context of

Hello! I am the creator of this experiment. Glad to see the conversation about hand tracking in the browser over here. This demos is done under the context of a series of creative experiments on how to use real time hand tracking in the browser for creative interactions. Will be posting more experiments soon. Tech background: I am using MediaPipe to control the hand rig in threejs. MediaPipe provides landmarks that are used to control a threejs Skeleton (hierarchy of bones with rotations). Feel free to ask, I will answer any questions!

jimmySixDOF 4y ago
This is amazing I had a browser plugin called flutter years back that was able to do webcam gesture recognition for scroll and forward/back. This is using threejs so I wonder how much is CPU or GPU and also how well this could, now or in the future, run under the hood in the background of a web game (or WebXR!) just as the input device and without too much overhead. Great Proof of Concept !!
- jackbach 4y ago
  Thanks for the nice words! Your plugin sounds like fun. In terms of using hand tracking for web games: my next experiments will use this setup to interact with 3D scenes.
  - jimmySixDOF 4y ago
    Looking forward to where you take this, and just to clarify I was just an end user of Flutter but was checking for a link (I had this installed years back) and turns out they got aquired by Google[1] so could be somewhere in Lense for all I know! [1] https://www.searchenginewatch.com/2013/10/03/google-acquires-flutter-creator-of-hand-gesture-recognition-technology/ https://www.searchenginewatch.com/2013/10/03/google-acquires...
marban 4y ago
Based on the sample at https://google.github.io/mediapipe/solutions/hands https://google.github.io/mediapipe/solutions/hands — that doesn't even sound all too complex.
- jackbach 4y ago
  I thought the same until I tried! As a matter of fact, lots of people over twitter have been sharing their frustration when they attempted to do the same thing. This is the funniest one: https://twitter.com/isjackwild/status/1617559339891396619 https://twitter.com/isjackwild/status/1617559339891396619 Here's some comments on my implementation: https://twitter.com/SketchpunkLabs/status/1617586619703230492 https://twitter.com/SketchpunkLabs/status/161758661970323049...
mncharity 4y ago
> creative interactions Fwiw, some things I've found fun: Clip-on fish-eye lens, intended for phone but fitting on laptop, for expanding webcam field of view. Additional cameras: on sticks above screen tips for high-res stereo positioning over kbd; asymmetric high-off-to-side to trade some resolution for some field of view (meh); high-overhead for whole-workspace tracking. Binocular periscope with webcam splitter and screen-tip mirrors (blech - low-res awkward fiddly). Look-down mirror on webcam, partial or full, to get kbd view (nice in VR). Look-down with curved mirror along top of keyboard to get "out along kbd surface view" and crufty touch detection for kbd-as-touch-surface (cute but fiddly - only makes sense to save a camera or two; caveat I had high-contrast white hands on black thinkpad kbd). Putting tracking markers on fingers (flats, a-frames, or cubes on velcro rings) makes for less jittery tracking, but is awkward (meh). Markers taped around keyboard help with calibration. Magic wand. I found I could more-or-less manage to type while holding a chopstick. So stuck a marker cube on one end, and an arc-sliced-off a small-Xmass-ball on tip, so it slides smoothly across (thinkpad) keys. Barber-pole rotation marker. Anvil'ed tip pressure sensor, a finger microswitch, and very thin and soft ribbon cable to arduino. But didn't actually get the pressure sensor working before punted on all this. Chopstick was narrow enough to avoid breaking hand tracking. Some gotchas: 2K camera resolution was painful for tracking. (Several years ago) mediapipe finger tracking was annoyingly noisy for doing stereo. You only get one usb2 camera per usb port, even if it's usb3 (maybe usb3 cameras allow working around that limit nowadays?). If you do hand, arm, face and marker tracking on several cameras, even with native gpu mediapipe, you're burning a lot of gpu just on the human interface device, before your likely-graphical-itself app even starts. If I had it to do over now, I'd punt mirrors, use 4K usb3 cameras, and at least with desktop, more cameras. Nicely merging high-latency camera tracking with lower-latency keyboard, touchpad, and graphics tablets, requires changes to the input event pipeline, and adapting apps to deal with "oh my! That space key pressed several keys ago - it was pressed with a pointer finger at position 3!, so that means we roll back app state and then ...". Here we are a half-century later, still banging on glorified xerox altos. We're so broken.
- jackbach 4y ago
  Thanks for the pointers! Do you have any video/media/recordings of your experiments?
p0w3n3d 4y ago
One use case I immediately thought for hand movement tracking like this is to help my disabled brother - tetraplegic - steer efficiently. Using mouse for him is sometimes too hard. Only in some cases. If one could use this as a macro launcher or more accurate joystick without attaching real joystick, this could help a lot
psychomugs 4y ago
I wish I had found out about MediaPipe before the tail end of grad school, but my collaborator integrated some neat stuff into our project. Very cool; thanks for sharing!

jimmySixDOF 4y ago

This is amazing I had a browser plugin called flutter years back that was able to do webcam gesture recognition for scroll and forward/back. This is using threejs so I wonder how much is CPU or GPU and also how well this could, now or in the future, run under the hood in the background of a web game (or WebXR!) just as the input device and without too much overhead. Great Proof of Concept !!

jackbach 4y ago
Thanks for the nice words! Your plugin sounds like fun. In terms of using hand tracking for web games: my next experiments will use this setup to interact with 3D scenes.
- jimmySixDOF 4y ago
  Looking forward to where you take this, and just to clarify I was just an end user of Flutter but was checking for a link (I had this installed years back) and turns out they got aquired by Google[1] so could be somewhere in Lense for all I know! [1] https://www.searchenginewatch.com/2013/10/03/google-acquires-flutter-creator-of-hand-gesture-recognition-technology/ https://www.searchenginewatch.com/2013/10/03/google-acquires...

marban 4y ago

Based on the sample at https://google.github.io/mediapipe/solutions/hands https://google.github.io/mediapipe/solutions/hands — that doesn't even sound all too complex.

jackbach 4y ago
I thought the same until I tried! As a matter of fact, lots of people over twitter have been sharing their frustration when they attempted to do the same thing. This is the funniest one: https://twitter.com/isjackwild/status/1617559339891396619 https://twitter.com/isjackwild/status/1617559339891396619 Here's some comments on my implementation: https://twitter.com/SketchpunkLabs/status/1617586619703230492 https://twitter.com/SketchpunkLabs/status/161758661970323049...

mncharity 4y ago

> creative interactions Fwiw, some things I've found fun: Clip-on fish-eye lens, intended for phone but fitting on laptop, for expanding webcam field of view. Additional cameras: on sticks above screen tips for high-res stereo positioning over kbd; asymmetric high-off-to-side to trade some resolution for some field of view (meh); high-overhead for whole-workspace tracking. Binocular periscope with webcam splitter and screen-tip mirrors (blech - low-res awkward fiddly). Look-down mirror on webcam, partial or full, to get kbd view (nice in VR). Look-down with curved mirror along top of keyboard to get "out along kbd surface view" and crufty touch detection for kbd-as-touch-surface (cute but fiddly - only makes sense to save a camera or two; caveat I had high-contrast white hands on black thinkpad kbd). Putting tracking markers on fingers (flats, a-frames, or cubes on velcro rings) makes for less jittery tracking, but is awkward (meh). Markers taped around keyboard help with calibration. Magic wand. I found I could more-or-less manage to type while holding a chopstick. So stuck a marker cube on one end, and an arc-sliced-off a small-Xmass-ball on tip, so it slides smoothly across (thinkpad) keys. Barber-pole rotation marker. Anvil'ed tip pressure sensor, a finger microswitch, and very thin and soft ribbon cable to arduino. But didn't actually get the pressure sensor working before punted on all this. Chopstick was narrow enough to avoid breaking hand tracking. Some gotchas: 2K camera resolution was painful for tracking. (Several years ago) mediapipe finger tracking was annoyingly noisy for doing stereo. You only get one usb2 camera per usb port, even if it's usb3 (maybe usb3 cameras allow working around that limit nowadays?). If you do hand, arm, face and marker tracking on several cameras, even with native gpu mediapipe, you're burning a lot of gpu just on the human interface device, before your likely-graphical-itself app even starts. If I had it to do over now, I'd punt mirrors, use 4K usb3 cameras, and at least with desktop, more cameras. Nicely merging high-latency camera tracking with lower-latency keyboard, touchpad, and graphics tablets, requires changes to the input event pipeline, and adapting apps to deal with "oh my! That space key pressed several keys ago - it was pressed with a pointer finger at position 3!, so that means we roll back app state and then ...". Here we are a half-century later, still banging on glorified xerox altos. We're so broken.

jackbach 4y ago
Thanks for the pointers! Do you have any video/media/recordings of your experiments?

p0w3n3d 4y ago

One use case I immediately thought for hand movement tracking like this is to help my disabled brother - tetraplegic - steer efficiently. Using mouse for him is sometimes too hard. Only in some cases. If one could use this as a macro launcher or more accurate joystick without attaching real joystick, this could help a lot

psychomugs 4y ago

I wish I had found out about MediaPipe before the tail end of grad school, but my collaborator integrated some neat stuff into our project. Very cool; thanks for sharing!