8 ms·
Handtrack.js: A library for prototyping realtime hand detection in the browser
- m3sh 6y agoif you are bald and your head is in the frame, it detects your head as hand.
- SftwreEngnr 6y agoSolution: grow your hair back
- VoidWhisperer 6y agoWhat was the score it determined when it detected your head as a hand? Might just be an issue of finetuning the minimum applicable score
- devxpy 6y agoI wonder if this problem can be solved by using the YOLO model in front of this to cut out the noise.
- vykthur 6y agoCurrent approach being explored to expand the training dataset to include faces/heads labels such that the model better discriminates this from hands.
- tgv 6y agoIt also recognizes my ceiling as a hand when there's no hand in front of the camera. Not enough negative training data? To satisfy the curiosity of VoidWhisperer: the score was 0.868. And it also sometimes loses track of a hand, in my case quite reliably when it's flat and seen from the side.
- mywacaday 6y agoI'm not bald and my head got a score of 0.9xx and my hand in the same frame was 0.7xx
- kreetx 6y agoNo, I think you should just stop fooling yourself and accept reality for what it is ;) (yes, I'm balding too and it's okay).
- amelius 6y agoReminds me of: https://gizmodo.com/ai-camera-mistakenly-tracks-referee-s-bald-head-instead-1845548314 https://gizmodo.com/ai-camera-mistakenly-tracks-referee-s-ba...
- vykthur 6y agoAuthor here, this is a known issue and being worked on for handtrack.js 2.0. The face/head errors is likely an artifact of transfer learning (handtrackjs is finetuned on the an object detection model trained using the coco object detection dataset which contains a person category). Current approach being explored to expand the training data to include faces/heads such that the model better discriminates this from hands. There are also size and speed optimizations being explored (smaller models such as efficientdet, quantization).
- kesor 6y agoHow is this better than the already available HandPose https://github.com/tensorflow/tfjs-models/tree/master/handpose https://github.com/tensorflow/tfjs-models/tree/master/handpo... ?
- enjoylife 6y agoA cursory glance shows the output of this project is simply bounding boxes for multiple hands, while the handpose project you mentioned is tuned for a single hand but it provides the key points for fingers and joints. I sense that handpose is probably far more useful for anyone going beyond a prototype.
- lazyjeff 6y agoI'm not the author for either project, but my impression from some tests has been that hand pose is obviously better information, but fickle. There's simply a limitation of how well a camera operating in the visible light range can do (especially if the palm is not facing the camera). If you don't need to match the image to a hand pose, then just tracking the hand position is going to be more accurate for that.
- apinstein 6y agoFacebook might disagree with your assessment... hand tracking research for VR has come a long way: https://research.fb.com/publications/constraining-dense-hand-surface-tracking-with-elasticity/ https://research.fb.com/publications/constraining-dense-hand...
- ofrzeta 6y agoI came to say that this will probably make our fan constantly spin until I saw that this is already in the TODO list of the project. Every client side feature such as background blur or face detection will make my (or every other person's I know) fans spin constantly so I will never use that feature. I don't know if there's a potential software way to solve this problem or do we have to wait until everyone's using an m2 or i8 cpu?
- gibolt 6y agoCamera resolution is likely to grow at the same rate as processor improvements. Downsizing stream first might be the best bet. Although, if this is meant for prototypes, it doesn't matter if you get to validate your idea's efficacy
- vanderZwan 6y agoImage size is already downscaled to 450 * 380, and afaik most ML libraries downscale the input by default. In that context a higher camera resolutions only really affect noise levels due to the effective super-sampling, not the performance of the library (assuming downscaling is an insignificant relative to the hand detection part).
- __MatrixMan__ 6y agoI'm not hip to the js world. Is there a common pattern for offloading the compute heavy parts of a js workload to a server somewhere? I'd totally just leave my pc running so it could do the heavy lifting for my mobile devices if that meant better battery life. If there's going to be waste heat, I'd rather it go into the building that I pay to heat.
- kreetx 6y agoThey don't need to be offloaded to a server, but just optimized: either running in web assembly or running on special machine learning cores (like on Apple devices).
- po1nter 6y agoIt detects my face as a hand with a score of 0.900+ especially when I tilt my head backwards.
- bathtub365 6y agoHow is something like this addressed in ML? In traditional software development there are tools like automated tests & static analysis to prevent your customers running into these types of embarrassing issues.
- vykthur 6y agoAuthor here, this is a known issue and being worked on for handtrack.js 2.0. The face/head errors is likely an artifact of transfer learning (handtrackjs is finetuned on the object detection model trained using the coco object detection dataset which contains a person category). Current approach being explored to expand the training data to include faces/heads such that the model better discriminates this from hands. There are also size and speed optimizations being explored (smaller models such as efficientdet, quantization).
- oksurewhynot 6y agoI used handtrack.js a few months ago to write an in browser theremin. It's not perfect but it allowed me to write the whole thing in about three hours and only kind of blew up my 2011 mbp. https://github.com/msmedes/theremin https://github.com/msmedes/theremin
- vykthur 6y agoHi, handtack.js author here, I am exploring some size and speed optimizations which might help with this (smaller models such as efficientdet, quantization). Will share once something is ready.
- tonydiv 6y agooh my. has anyone thought about connecting this to .def( 'sin', {U+1F6A6}.2Moj ).( {U+1F6A6}.2Moj} ).with<3( {U+1f9cf} | {U+fe0f} ); d-ratz moj nt spprtd. here are the unicodes. lrn mor: https://xn--7bi.hns.to/ https://xn--7bi.hns.to/
- rufus31415 6y agoWebXR is a JS API that, among other things, allows you to track hands using the device's capabilities. Handtrack.js is a good alternative for browsers that are not compatible with WebXR.
- villgax 6y agoA better model is BlazePalm inside MediaPipe/Google FOSS