5 ms·
Show HN: Controlling 3D models with voice and hand gestures
I'm sharing my project to control 3D models with voice commands and hand gestures:
- use voice commands to change interaction mode (drag, rotate, scale, animate)
- use hand gestures to control the 3D model
- drag/drop to import other models (only GLTF format supported for now)
Created using threejs, mediapipe, web speech API, rosebud AI, and Quaternius 3D models
Githhub repo: https://github.com/collidingScopes/3d-model-playground https://github.com/collidingScopes/3d-model-playground
Demo: https://xcancel.com/measure_plan/status/1929900748235550912 https://xcancel.com/measure_plan/status/1929900748235550912
I'd love to get your feedback! Thank you
- getToTheChopin 1y agoHere's a quick video demo showing how it works: https://x.com/measure_plan/status/1929900748235550912 https://x.com/measure_plan/status/1929900748235550912
- ActionHank 1y ago[flagged]
- deleted 1y ago[deleted]
- pvg 1y agohttps://xcancel.com/measure_plan/status/1929900748235550912 https://xcancel.com/measure_plan/status/1929900748235550912 for the tweetless
- xnx 1y agoSounds very cool, but I could not make sense of the on-screen instructions. Some images or animations would go a long way to explain the controls.
- getToTheChopin 1y agoSorry about that, the instructions need to be improved. Does this video demo help? https://x.com/measure_plan/status/1929900748235550912 https://x.com/measure_plan/status/1929900748235550912 If it makes it clearer, I'll upload it to the github repo directly
- xnx 1y agoThat video did help. I think I was thrown off by two things: 1) I was expecting 3D controls with more direct mapping (e.g. rotating my hand rotates the model). This is more like gesture mouse controls. 2) Some of the controls were too subtle. The scaling between my gesture size and effect on screen was smaller than I expected. Great area to develop though. There's so much untapped potential in applying Mediapipe.
- getToTheChopin 1y agoThank you for the feedback. I'll continue to work on it!
- ednite 1y agoGreat job! Looks very useful for interactive content creations and product showcasing. Definitely will testing it more. Thanks for sharing.
- getToTheChopin 1y agoyes I'd love to go further with this concept so that 3D / CAD designers could easily present their models during video calls. thank you!
- appendixv2 1y ago[flagged]
- nimzoLarsen 1y ago[flagged]
- deleted 1y ago[deleted]
- atuladhar 1y ago[flagged]
- thomashop 1y ago[flagged]
- corysama 1y agoI understand you need your face in the videos for the demos. But, want to mention that you should make sure your system works with your hands in your lap. As shown, the user is going to experience "gorilla arm" fatigue very quickly.
- getToTheChopin 1y agoGood points, maybe a second camera (phone?) pointed downwards at the tabletop would be good for that. Then the user can rest their hands in a "normal" position. Thank you for the feedback!
- doakes 1y agoVery cool! I like the different modes. I've always been fascinated with this space and products like Leap Motion: https://www.youtube.com/watch?v=zXghYjh6Gro https://www.youtube.com/watch?v=zXghYjh6Gro
- getToTheChopin 1y agoIt's a shame that computer vision tech like Leap / Eyetoy / Kinect didn't have lasting power. So much fun to build with
- danvoell 1y agoAwesome, nice work! This type of tech opens up a world of physical games.
- getToTheChopin 1y agoI made a demo game where you need to dodge the evil bouncing DVD logo by moving your body: https://x.com/measure_plan/status/1924830500541157570 https://x.com/measure_plan/status/1924830500541157570 I'm working on a couple other body movement concepts and hope to share soon :)
- hulium 1y agoSee also Webcam Mania: https://webcam.sulat.net/ https://webcam.sulat.net/ It uses a bit simpler approach, only detecting movement, but it works good enough for such games.
- Greta4Gaza 1y ago[dead]
- ugh123 1y agoAmazing! Maybe use specific finger positions/gestures to trigger a rotation and scale functions (index finger up and within a bounding box of the model perhaps for rotation, similar for pinch to fingers to scale).
- getToTheChopin 1y agoI'll try it, thank you! I separated them into completely different interaction modes to avoid misfires, but there's definitely room for efficiencies
- Ey7NFZ3P0nzAe 1y agoI would be surprised if sign language didn't have an efficient way to convey digits.
- chimpansteve 1y agoSlightly on topic - anyone remember LeapMotion and is anyone aware of any current support for that? Found an original one in a drawer when I was having a clearout the other day
- one_more_q 1y agoI have one as well! Sorry no idea what it can be used for nowadays