10 ms·
Show HN: Open-source digital stylus with six degrees of freedom
- extraduder_ire 3y agoOutside tracking with a camera is not something I would have thought of. Seems cool. Reminds me of how sad I am nobody's done a good job of cheaply cloning the lighthouse tech that valve/htc use.
- jcparkyn 3y agoI should point out that I'm not the first person to use camera tracking for this [1], but to my knowledge there hasn't previously been a serious attempt to combine it with inertial or pressure sensors (which are both necessary for competing with graphics tablets) or make it open-source. [1] http://media.ee.ntu.edu.tw/research/DodecaPen/ http://media.ee.ntu.edu.tw/research/DodecaPen/
- two_handfuls 3y agoIsn’t camera tracking + inertial sensors exactly how the Oculus Rift headset & controller tracking worked?
- jcparkyn 3y agoYes (or something like that, I don't know all the details), but they use infra-red light which requires a dedicated camera. I think they might also require more than one camera (although it is possible to do monocular IR tracking).
- mtsr 3y agoAFAIK you can actually get the sensors used for tracking at somewhat OK prices. But iirc the sensors actually do some part of the position calculations and as such aren’t simple enough to be really cheap.
- r2_pilot 3y agoYep, they're actually just tiny solar panels! I got several trying to figure out how to use the lighthouse system. Not enough hours in the day and at the time there wasn't enough compute in a small package cheap enough for me.
- charcircuit 3y agoToo bad this wasn't made as an OpenXR API layer so that it could be used with existing software
- noddingham 3y agoIt's open source so...
- ipsum2 3y agoThe great thing is that its open source, so you can add whatever niche APIs you want to it!
- charcircuit 3y agoWhy would I be the one to do that? I already have a pressure sensitive 6dof stylus that is compatible with openxr. OpenXR isn't a niche API when you are dealing with 6dof input devices.
- cududa 3y agoI know this might be shocking, but OP didn’t make this for you. There’s actually all sorts of people out there making all sorts of things without considering your specific needs
- charcircuit 3y agoWhy are you using that tone? I never had the intention to use this. I am just stating that there is a missed opportunity that exists. Why are you implying that I want this as feature for me to use?
- jcparkyn 3y agoI'll chime in and say that OpenXR integration is a good suggestion, just not something I had the time/effort to implement (this is a university project, after all, and was focused more on the core tracking tech). I think people just took issue with the way you phrased your original comment.
- slaucon 3y agoThe rolling shutter compensation is pretty cool and isn't something I would have thought of. Did you know that would be an issue from the start or notice it only after you built the rest of the system?
- jcparkyn 3y agoI knew it would have an effect (most of the literature for similar projects just uses global shutter cameras for this reason), but wasn't sure how significant it would be. It turned out to be small enough that it usually wasn't super noticeable, but in certain cases it really showed up (e.g., rotating the pen while keeping the tip in one place). The thing I was most surprised by was how effective my solution was, given that it's a pretty gross approximation of reality. There are lots of much more sophisticated techniques for dealing with it, which I didn't end up needing. One thing I would've liked to try out is using rolling-shutter-aware PnP [1], which can theoretically estimate pose and velocity simultaneously from one image, by exploiting the rolling shutter distortions. [1] https://www-sop.inria.fr/members/Philippe.Martinet/publis/2006/eccv06omar.pdf https://www-sop.inria.fr/members/Philippe.Martinet/publis/20...
- ipsum2 3y agoVery cool! It has the added benefit of being able to manipulate objects in 3d. How does it compare to graphic tablets in terms of accuracy?
- jcparkyn 3y agoCurrently it's not quite at the level of graphics tablets, but it's not too far off, and I think there's quite a bit of potential to improve it using similar techniques [1]. In terms of absolute accuracy, I measured an average error of 0.89mm (for the position of the tip) across the entire area of an A4 page with the camera in one place. In practice you have more precision than that though, because most of the errors are constant biases (not random noise). For example, here's one of the tests [2] I did for the thesis, which compares the recorded stroke to what I actually wrote (scanned from carbon paper). After aligning the two captures (as a global 2D position offset, everything else is retained), the average distance from the recorded stroke to the scan was 0.158mm. [1] This paper (which I linked in another comment) uses some more advanced techniques for pose estimation which could definitely be applied here (but it's closed-source, and I didn't have time to re-implement it from scratch): http://media.ee.ntu.edu.tw/research/DodecaPen/ http://media.ee.ntu.edu.tw/research/DodecaPen/ [2] https://github.com/Jcparkyn/dpoint/files/13329235/main-sketch-plot.pdf https://github.com/Jcparkyn/dpoint/files/13329235/main-sketc...
- moritonal 3y agoBrings back joyful memories of me begging random Professors from around the world over email for the source code of their papers.
- abdullahkhalids 3y agoNice. It seems there is a systematic error towards the edges of the paper, which can easily be fixed in software. Though it would be nice to do more tests. For example, there is some error when drawing sharp corners. It would be instructive to see how that changes as you change the angle, etc. And how that changes based on location on the paper.
- orbital-decay 3y ago
- tomcam 3y ago> This project was part of my undergraduate thesis for electrical engineering. I Undergrad! If you didn't get top marks on this there is no justice
- jcparkyn 3y agoThanks! I'll get results in about two weeks (fingers crossed)
- an_aparallel 3y agouni marks != quality :-) I agree with you though.
- cududa 3y agoI scrolled down to the picture first and the desk made me instantly think “ah, a dorm room”
- master-lincoln 3y agoWhy do you think so? When I was doing undergrad, the emphasis was on scientific work. So novelty or applicability were not important for grades, but if your written communication about it was scientific. I didn't see a write-up of this project, so I couldn't judge it grade wise
- Naracion 3y agoI agree, and this is one of the things that are problematic with academia. Not that scientific rigor / work / dissemination shouldn't be rewarded--it should. But so should work like OP's.
- master-lincoln 3y agoundergrad != academia Undergrad is trying to learn the necessary basics to be able to get into academia. Of course undergrads could publish papers already and be part of academia, but in this case I am not sure if there is any novelty or if it's just a nice product idea.
- deleted 3y ago[deleted]
- bloopernova 3y agoVery cool. They've done what I've daydreamed about, and actually got it to work! When I played Elite: Dangerous I used a "hand on throttle and stick" (HOTAS) setup, along with foot pedals. I couldn't help but think that there must be a better way to control a spaceship: your ship can pitch, yaw, and roll in addition to being able to fire thrusters in 6 directions. I wanted a handheld ship model that I could move such that the ship in Elite would move in the same way. The linked project looks like it could do just that. Thrust would be controlled in a similar way, but with my other hand. Strange or new input models like that are so amazing to me. Our imagination can really fly high with these sorts of capabilities.
- squigz 3y agoThis is why 2 sticks is a fairly common setup for space games
- Arelius 3y agoI still hold onto my SpaceOrb369 for this sort of game. There is a guy that makes a converter to make it work over USB... https://en.m.wikipedia.org/wiki/SpaceOrb_360 https://en.m.wikipedia.org/wiki/SpaceOrb_360 https://www.tindie.com/products/vputz/orbotron-9001-version-4/ https://www.tindie.com/products/vputz/orbotron-9001-version-...
- myself248 3y agoI have the slightly newer version, branded by HP as the SpacePilot, and use it daily in Fusion 360 CAD. It has a native USB connection, and 6 user-definable hotkeys below an unimpressive LCD. It requires some old drivers that aren't officially supported (why would they remove support for perfectly good hardware from the newer drivers? To send good stuff to the landfill, of course!), but when Autodesk tried to move Fusion to the new driver model exclusively, user outcry persuaded them to leave the old drivers in as an option. Apparently there's quite a few of us using those SpacePilots, and the phrase "from my cold, dead fingers" comes up not infrequently.
- w-ll 3y ago
- lagrange77 3y ago1. Very cool project 2. Helpful documentation 3. Nice real world example for the use of a Kalman Filter!
- jimmySixDOF 3y agoI'm a big fan of all things 6DOF ! Nice work on the hardware and computer vision pose work but I am almost more impressed by the software surface you are drawing into and able to rotate. Thats interesting and could be used with any tangible user interface control like a finger slider for the same effect. Good project for problem solving skills looks like you nailed it bravo! Btw the first 6DOF controller I had other (than a hacked WiiMote controller as a ir led Bluetooth receiver [1]) was the logitec mx air which was ahead of its day [2]. [1] https://web.cs.ucdavis.edu/~okreylos/ResDev/Wiimote/MainPage.html https://web.cs.ucdavis.edu/~okreylos/ResDev/Wiimote/MainPage... [2] https://www.cnet.com/reviews/logitech-mx-air-review/ https://www.cnet.com/reviews/logitech-mx-air-review/
- tomp 3y agoVery cool! Could be useful for robotics / VR as well. One-camera hand tracking anyone? Question: could you use gyro+accel to track pressure as well? Or at least "taps"? Another question: how much does it cost? in particular, the pressure sensor...
- jcparkyn 3y ago1: You could absolutely use gyro/accel for detecting taps, but for proper pressure sensitivity (i.e., changing pressure in the middle of a stroke), there's not much you can do except have a pressure sensor. It's theoretically possible with a sufficiently accurate pose estimate and a springy pen tip, but not feasible at the level of accuracy I got. 2: I paid about $20 AUD for the pressure sensor, but they can be had for quite a bit cheaper (~$5 USD) in the US (https://www.arrow.com/en/products/hsfpar003a/alps-electric https://www.arrow.com/en/products/hsfpar003a/alps-electric). Only problem is they're quite specialized, so not many places sell them. The custom PCB was another $10AUD, and the Arduino was about $20. There's a full parts list at https://github.com/Jcparkyn/dpoint/blob/main/setup-guide.md https://github.com/Jcparkyn/dpoint/blob/main/setup-guide.md.
- tomp 3y agoanother question: how do you estimate _depth_ - i.e. the dimension _away_ from the camera? I'm guessing it's the size of the ArUco markers, combined with accelerometer for smoother tracking - but that seems quite imprecise?
- jcparkyn 3y agoThat's done by the PnP solver, but yes it's essentially looking at the size of the aruco markers on the screen. This is implicit in the equation the PnP solver is trying to solve: "find the pose (position and orientation) that results in the minimum re-projection error for all the marker corners". Re-projection error is the difference between the observed 2D location of each marker corner, and the theoretical 2D location for a given pose (using standard camera projection equations). You're right that this is the least precise dimension for PnP, but it turns out to be good enough if the corner positions are decently accurate. Using sub-pixel refinement for the corner locations helps a lot (this is built into OpenCV — it looks at the brightness values for multiple pixels around the corner to get an estimate more accurate than one pixel). Having corner positions further apart in 3D space helps as well, which is part of the reason I used two "rings" of markers. In my case I was getting somewhere in the ballpark of 1-2mm of depth error at ~3cm away from the camera, so ~0.5% relative error. Smoothing from the accelerometer stops this from being noticeable most of the time, but it will sometimes crop up in bad conditions (e.g. if one of the markers is being missed on some frames due to lighting problems).
- crazygringo 3y agoVery cool. The use of a webcam really makes me wonder if there's a future where our regular single ~78° FOV webcams are going to be replaced by dual (stereo) fisheye webcams that can: - Enable all sorts of new UX interactions (gestures with eye tracking) - Enable all sorts of new peripheral interactions (stylus like this, but also things like a steering wheel for racing games) - Enable 3D 180° filming for far more flexible webcam meetings, including VR presence, etc. The idea of being able to use the entire 3D space in front of your computer display as an input method feels like it's coming, and using a webcam the way OP describes feels like it's a little step in that direction.
- moritonal 3y agoI'm sadly bearish on this kind of stuff. The Mac touch-panel showed that even a context-aware, reactive and fully integrated input struggled to break the flow gained by keyboard+mouse. Even today some people are still on the fence about the new fangled mouse!
- two_handfuls 3y agoThe Mac touch bar sucked, that why it didn’t gain traction.
- moritonal 3y agoLikely valid, but can you say why it sucked? And why any other input wouldn't struggle with the same faults?
- two_handfuls 3y agoIt sucked for the following reasons: - too easy to accidentally touch while reaching top-row keys. - cannot be used by feel. - replaces existing keys, removing functionality. - turns off sometimes, so we have to tap once to wake it first. - some common actions used to be 1 key, are now two taps (volume, brightness). - does not always register my taps. No unambiguous indication that the tap was recognized.
- mncharity 3y agoNice. Especially the fusion. Some random-ish thoughts from exploring "a laptop keyboard... with hand pose, 3D stylus, and touch". Adding buttons yields a 3D mouse - but camera coverage can be a pain. Note the body is largely empty (and battery could be slimmed) - I could more-or-less type while holding a slim chopstick/toothbrush-like stylus (even with the markers, and a weirdly big tip). A big tip (sliced from a ping-pong ball sized xmas decoration) could slide fairly smoothly on a ThinkPad keyboard (and gave room for a less compact force sensor, and an extra tip marker). Thin stranded silicone ribbon cable can be string-like flexible - I just tethered the stylus to an arduino to get started. Hmm... I wonder what the inertial sensor might make of something like a dimpled metal clicker, as a button (or three)?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- sshkey 3y agowow, very impressive project! really I love it, I will try to build it one definitly.
- ezconnect 3y agoAdd a laser range finder on the camera and retroreflector on the pen and we have a cheap implementation of a 3D measuring device.
- wnolens 3y agoThis is really cool, and right out of my dreams. Since the move to full remote work at my company, I've been longing for whiteboard sessions. The best I could conjure up was to use an old iPad and load up a shared whiteboarding webapp in a browser on both iPad and my desktop, and then share screen from my desktop, using a cheapo stylus on the iPad to draw. It's pretty good, but my iPad is super old and slow. Replacing it is not affordable. But a conventional web cam + a relatively low-tech stylus would be much better, and drop the need for an external device completely.
- abdullahkhalids 3y agoThe cheapest Wacom tablet [1] is 67CAD. Other brands have cheaper ones. [1] https://wacomstore.ca/product/one-by-wacom-small/ https://wacomstore.ca/product/one-by-wacom-small/
- sj4nz 3y agoFantastic. I love that this is brand-independent, like buying a mechanical keyboard.
- joshuahhh 3y agoImpressive work! I wonder: is the inertial tracking sufficient to cover for occasional occlusion of the markers?
- jcparkyn 3y agoThat depends what you consider "occasional", but for most definitions the answer would be no. I'm sure there are better ways to handle the inertial measurements than what I did, but even so, double integration drift causes errors very very quickly. I doubt it'd be possible to go over a second, and my implementation struggled with anything over ~150-200ms. However, it does an impressively good job with low camera frame rates. I tested it at 10FPS (discarding every third frame), and the results are barely distinguishable from 30FPS. Below that it starts having difficulty.
- denton-scratch 3y agoTFA mentions a gyroscope, but I don't see one in the assembly diagram.
- guhidalg 3y agoI believe the PCB has the IMU built into it along with the BT module. https://wiki.seeedstudio.com/XIAO_BLE/ https://wiki.seeedstudio.com/XIAO_BLE/
- piotr123 3y ago[flagged]
- piotr123 3y ago[flagged]