5 ms·
Handsfree.js – integrate face, hand, and/or pose tracking to front end projects
- albybisy 5y agothe library and the demos are not loading for me.
- twobitshifter 5y agoWhat are some of the applications for this?
- Gys 5y agohttps://github.com/midiblocks/handsfree#examples https://github.com/midiblocks/handsfree#examples
- danielskogly 5y agoI found this demo of him playing "Into the Breach" with only Face Gestures very impressive! https://twitter.com/GoingHandsfree/status/1400221710838747136 https://twitter.com/GoingHandsfree/status/140022171083874713...
- umvi 5y agoYou could integrate this with adtech to help make advertisements more interactive and to analyze how the viewer is receiving the ad by their pose and facial expressions. Based on that you could have an algorithm serve only ads that improve facial expressions and poses to positive ones
- Solocomplex 5y agoAnother excellent reason to disable JavaScript in your browser
- jcdietrich 5y agoCould play punch the monkey!
- lolinder 5y agoI hope we haven't yet gotten to the point where people will mindlessly click "Yes!" when a random website asks for webcam access.
- notjustanymike 5y agoWebXR comes to mind.
- blensor 5y agoI can't speak about this specific project but pose tracking in the browser can actually be really useful to get it on mobile without the need for dedicated apps. We are currently experimenting with using phones (or tablets or basically anything with a browser and a camera) as a tracking source for full body tracking on the Oculus Quest ([1]) and being able to just use a Tensorflow model in the browser has cut down the time of development on the mobile device side to almost nothing. There are of course a whole lot of other issues one buys into with that approach (browser security policies, the whole thing not working in certain countries due to problems with loading the model) but that's manageable [1] https://www.youtube.com/watch?v=zz9S5hgrWpM https://www.youtube.com/watch?v=zz9S5hgrWpM (relevant part starts at 0:29)
- tppiotrowski 5y agoI made a slouching detector a while back. It calibrated the distance of your face from your laptop and if you start leaning in it’ll give you a notification to sit back. It’s a bit dated but still works on Chrome and FF, I believe: https://slouchcam.com https://slouchcam.com
- ranguna 5y agoWow this is amazing! I remember a few years ago in my thesis I was trying to use something similar in ROS to detect hand gestures and make a robot drive toward you, it's incredible to see this in the browser!
- GoingHandsfree 5y agoOh that would be so epic! I did something similar during my residency for Handsfree.js but for "puppeteering" a robot: https://giphy.com/gifs/1XE2rnMPk6BFu8VQRr https://giphy.com/gifs/1XE2rnMPk6BFu8VQRr
- rgovostes 5y agoOn Safari for Mac, you must enable Develop → Experimental Features → WebGL 2.0.
- gregsadetsky 5y agoI'll use this post / opportunity to ask the HN community (I understand that this is a technically complicated, and potentially sensitive subject) I've been approached by a few people who are doing what they can to lower the cost of prescription glasses. It's a mix of (licensed) ophthalmologists, people interested in offering community health services, etc. What is the current state of the art for pupillary distance (PD) measurement based on face detection / pupil tracking using laptop webcams or mobile (front) cameras? I imagine (obviously?) that there must be scholar research on the acceptable error margins for a PD measurement (depending on the type of vision condition, i.e. farsightedness, etc.)? Would using something like Handsfree.js or https://github.com/esimov/pigo https://github.com/esimov/pigo (which has pupil detection) be a good start, or would these be an ~order of magnitude off in terms of the necessary margins? Thanks a lot.
- notum 5y agoIt comes down to physics. Cameras and their lenses come in all shapes and sizes, most of webcams don't expose their geometry in a programmatically accessible way, even if they did, most of them don's have means of determining distance to the subject. We can detect faces, eyes and pupils, but with no reference points we have no idea how big or far apart they are.
- gregsadetsky 5y agoThanks! What about sites (such as Warby Parker) that make you hold up a credit card (which has a standard size) and use that to determine (via what I assume is a rule of three) the PD? [0] I can see of course problems with the card not being “perpendicular” and other 3d distortions. [0] - https://www.warbyparker.com/pd/instructions https://www.warbyparker.com/pd/instructions
- mncharity 5y agoFwiw, there's https://www.eyeque.com/pdcheck/ https://www.eyeque.com/pdcheck/ . And Google's MediaPipe Iris used iris diameter for distance estimation.[1] EyeQue is perhaps a cautionary tale. "Change the world" "few dollars of optics + phone = D+CYL" become $45 one-person-per-subscription "vision tracker". [1] https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...
- esens 5y agoI have tried the underlying library, MediaPipe/TensorFlowJS, and it is only okay. Once you try to use it for visual augmentation, I found it doesn't track that well and it has problems with scale. So, while it may look good with the skeletons, notice they do not have useful demos, mostly just proof of concept demos that do almost nothing.
- blensor 5y agoHave you tried the newest MoveNet from google? That thing works really well. I only started experimenting with the Lighting model (not Thunder) for our VR game but even that is pretty robust. If you want to see how stable it is, have a look at that: https://youtu.be/zz9S5hgrWpM?t=147 https://youtu.be/zz9S5hgrWpM?t=147 It's tracking me with the Oculus Quest on the head, so it does not even have facial features to go on
- jonplackett 5y agoThanks for the link - Does MoveNet give you 3D positions / rotations for each joint or just a 2D position on the image?
- blensor 5y agoJust the 2D positions plus a confidence level. In the video we are using only one view so it's no true full body tracking yet. We are currently trying to use two views to get a rough 3D tracking. The good thing is that the positions of the detected points are relatively stable so with a big enough baseline that should work out for a rough 3D pose estimation
- villgax 5y agoBlazePose does give 3D portions for non-facial key points
- GoingHandsfree 5y agoI'm very sorry you don't think Handsfree.js has useful demos. That kind of hurts because I put a great deal of effort into making them useful as starting points (I started all this while I was homeless to help someone). I work on Handsfree.js full time and my goal is really to create simple demos so that others like yourself can run with it! Maybe you'll find some of these more useful? - Face Pointer, and accessibility tool to help people who can't use a mouse/keyboard: https://handsfree.js.org/ref/plugin/facePointer.html https://handsfree.js.org/ref/plugin/facePointer.html - Palm Pointers, scroll the page and even multiple scroll areas at once: https://handsfree.js.org/ref/plugin/palmPointers.html https://handsfree.js.org/ref/plugin/palmPointers.html - Gesture Mapper, map static poses in seconds: https://handsfree.dev/tools/gesture-mapper/ https://handsfree.dev/tools/gesture-mapper/ - Face Coding (not ready), to help people with disabilities code by snapping blocks together: https://twitter.com/GoingHandsfree/status/1400560152319729664 https://twitter.com/GoingHandsfree/status/140056015231972966... - The library itself doubles as a Chrome Extension, here are some examples of what you can do with it: https://handsfree.dev/sites/ https://handsfree.dev/sites/ I have dozens and dozens of more experiments. Most of them are pretty basic, but the point is to show you what's possible :)
- bjornstar 5y agoThere's a nice demo of using handsfree.js to drive a car in a racing game -- https://twitter.com/GoingHandsfree/status/1400322333013798914 https://twitter.com/GoingHandsfree/status/140032233301379891...
- amelius 5y agoHis comment: The controls are whacky as hell and it took 5min of practice but it's just a tech demo ...
- GoingHandsfree 5y agohaha yeah, the idea is to do one radically different use case every single day for 100 days straight so I don't always have time to make them smoother I did do another whacky one today that is a lot easier to do. It's called "Flintstone Mode": https://twitter.com/GoingHandsfree/status/1401366484920590336 https://twitter.com/GoingHandsfree/status/140136648492059033...
- bjornstar 5y agoHe just posted another demo where you kick your feet to accelerate a la The Flintstones -- https://twitter.com/GoingHandsfree/status/1401366484920590336 https://twitter.com/GoingHandsfree/status/140136648492059033... Pretty fun to watch!
- FabianCarbonara 5y agoVery cool project! This looks like so much fun to build. Thank you for the great documentation of your endeavour!
- powerslacker 5y agoProbably the coolest library I've seen in a while.