10 ms·
FingerIO: Using Active Sonar for Fine-Grained Finger Tracking
- jbverschoor 11y agoFake video.....
- halotrope 11y agoThis looks interesting. Can you try it somewhere?
- lucb1e 11y agoDoesn't seem like it. Until anyone other than the researchers tried it, I don't really believe it works outside of controlled environments.
- deleted 11y ago[deleted]
- sarreph 11y agoI love it when people find ways of using existing hardware with software innovation to make new interactions such as this!
- faded242 11y agoUhh.. not the best name choice in my opinion.
- shi 11y agoYep. When I read the title I thought it was a parody on IO-names.
- basicallydan 11y agoVery cool, good job :)
- IanCal 11y agoReally interesting. Reminds me of SOLI (which is radar rather than sonar): https://www.youtube.com/watch?v=0QNiZfSsPc0 https://www.youtube.com/watch?v=0QNiZfSsPc0 Is there a way of trying this out? I know it'd only be demo line drawing applications but it'd still be interesting to try.
- 4684499 11y agoThis is much like Project Soli. I started dreading devices like that. Sonars everywhere.
- gr3yh47 11y agoFirst thing i thought of is the horrible privacy implications when this tech becomes ubiquitous and cloud based
- debarshri 11y agoHow about security? How does it protect other uses from controlling your device?
- DuckyC 11y agoHow does your phone protect other people from pressing your volume buttons, or using your touchscreen? Security is pretty irrelevant for a HID.
- gr3yh47 11y agoit's obviously pretty different when you don't have to be touching the phone to interact with it
- njharman 11y agoNot when it's a few inches away. It's less distance than voice interaction.
- lucb1e 11y agoA lot of people are a few inches from my pocket every day. And I can easily imagine software listening for such sonars and sending back fake responses. I think debarshri has a valid point.
- njharman 11y ago> A lot of people are a few inches from my pocket every day. Really? And your phone is on? With the screen visible for them to see what they are interacting with. And as they are fumbling around in your personal space inches from your hand or pocket, no one notices. Get real! If true you should be more worried about pickpockets than some random gestures.
- njharman 11y agoNot when it's a few inches away. It's less distance than voice interaction.
- xuan 11y agovery interesting!
- magic_man 11y agoI work with sonar and the physical positioning of the sensors is important in trying to get useful results. Why is it these academic types don't release the apks or software? Just publications and maybe a video.
- mattlutze 11y agoYou should try asking these academic types for the data and source code. A lot of researchers are more than happy to discuss their work, but a big part of the academic industry is your research's impact and references. A way to get a better handle on who is looking at or following on with your work is to implicitly ask them to have a conversation with you before getting the whole kit 'n caboodle.
- Agustus 11y agoAdditionally, a number of these research programs take the source code and make a go of a small business product.
- guyzero 11y agoAnd in many cases the universities own the IP associated with the research that goes on in their departments so they keep the source and treat it like a company would treat a trade secret. It's likely Washington will patent this and try to license the patents.
- CamperBob2 11y agoI think we left the topic of "science" about 4 posts up. I don't know what's being described here, but it's not science. Yet somehow, I get the feeling that I'm paying for it.
- sorenjan 11y agoWhat's the legal situation with these academic research papers? Am I allowed to implement an algorithm from a research paper and then either sell the software or release it as open source? I assume other academics are allowed to reimplement methods in order to reproduce the result and to compare to their own methods. Can I do the same as a learning exercise?
- adrianN 11y agoI wonder how much power all the processing draws. Judging from the slow movements and the delayed update on the screens in this video, it's pretty heavy on the processor.
- jerf 11y agoThe question is, is the multisecond latency because of processing and code efficiency limitations for a academic research project, or is it because the data is unusable without two seconds of smoothing? Given what I see in the video I could argue either way. I do note the latent signal is still a bit noisy, but then, touch screen input isn't necessarily clean either. But it's also worth keeping in mind this is all off-the-shelf hardware. It seems very likely to me that if a cell phone or smart watch was designed to do this from the get-go that several easy hardware improvements and maybe a bit of custom DSP work would make this work much better. (By "easy hardware improvements", I mean things like speakers intended to emit frequencies for sonar, microphone arrays intended to receive them, etc.) From that perspective, even if the system we saw is fundamentally limited I'd still call it incredibly promising considering the constraints it is operating under! If I were a smart watch manufacturer I'd be falling over myself to get one of my best engineers and one of my best recruiters an appointment with these people.
- foobarqux 11y agoThey are calculating autocorrelation and a Fourier transform so they need to buffer the data. Two seconds is probably the shortest buffer that works reliably.
- mattb314 11y agoThe accompanying paper[1] claims the phone lasts four hours running the current version of fingerIO, but also that improvements could be made to preserve power (such as reducing the sampling rate). [1] http://fingerio.cs.washington.edu/fingerio.pdf http://fingerio.cs.washington.edu/fingerio.pdf
- yread 11y agoI wonder if you can just run it on any smartphone or you need to configure the positions of microphones beforehand
- xtf 11y agoCats will love it. Thats why the first ultrasound remote never became standard
- viewer5 11y agoWhat do you mean?
- mcculley 11y agoI suspect he means that this would be audible to cats and have unintended side effects.
- mxuribe 11y agoAww, this is just going to wreak havoc for whales and dolphins once divers start using these! ;-) https://en.wikipedia.org/wiki/Marine_mammals_and_sonar https://en.wikipedia.org/wiki/Marine_mammals_and_sonar
- supergirl 11y agocool idea but the tracking will never be good enough to be practical. if people really want this type of interaction then phones will start to incorporate specialized hardware for it.
- jstapels 11y agoI wonder how accurate it really is. The demo video didn't match up with the movements at all and the on-screen drawings looked like prerecorded video that they were trying to sync to. It's a neat idea, but without a dedicated component or an extremely high-speed RTOS, you're not going to come close to the level accuracy that's really needed to do the math and still allow interaction. I don't mean to rain on the parade, but I just don't think they really have anything usable.
- noir-york 11y agoIf the tech is great then wonderful, but I still remember LeapMotion...
- bossx 11y agoHave you tried the Orion SDK? It's an order of magnitude improvement in tracking accuracy, even with the older hardware. https://developer.leapmotion.com/orion https://developer.leapmotion.com/orion
- alexwebb2 11y agoI tried the "massive improvement" they released before Orion and it was still terrible. Only so many times a company can say they've got their issues ironed out before I stop believing them.
- LesZedCB 11y agoOrion really is quite impressive. I tried the improvement as well. Orion actually works.
- ajacksified 11y agoHmm, I'll have to try it out tonight. I have some 4-year-old v0.6 and v0.8 hardware in a drawer somewhere.
- MasterScrat 11y ago
- JamesBaxter 11y agoI wonder how well it would work in a noisy environment?
- labithiotis 11y agoIsn't it too early for April fools ?
- ape4 11y agoExactly my thoughts. I was watching the video thinking this is a bit far fetched. Just like when I learned about Google's moon base.
- wyldfire 11y agoI'm looking forward to the first theremin app.
- tootie 11y agoI assume it would be impossible because the sound of the music would interfere with the sonar?
- hardwaresofton 11y agoWouldn't the music only be on the audible spectrum? Sonar works (can work?) on the inaudible spectrum.
- fudged71 11y agoHeadphones
- fitzwatermellow 11y agoThis is cool. Thinking about smartphones as sensors opens up so many possibilities, even if their capabilities aren't nearly as accurate as dedicated devices. Wondering if the sonar information can be combined with images from the camera to create a close-range depth camera?
- k_bx 11y agoWe need to immediately improve on PIN-code protection upon cash-withdrawal in ATMs. The problem has been there for a while, but man, it gets easier and easier.
- monatron 11y agoSorry, I'm not sure how this is relevant or if this sentence even makes sense. Can you explain?
- andrew3726 11y agoI guess he means that you can place a smartphone near an ATM and catch the PIN-code with this technology somehow.
- cafeoh 11y agoI suppose he means that a small microphone/speaker setup near an ATM machine's keypad could allow you to track the position of the finger and get the corresponding keypresses without any intrusive modification to the ATM itself. Of course that would only be valid if you proceed to then steal the person's card right afterward, so all in all, not that useful.
- seszett 11y agoIsn't it easier to just put up a small camera? Then you can also record the card's number if you fail to steal it afterwards.
- swiley 11y agoIt would be easy to build a device that did something similar to this and captured pin codes on ATM machines.
- deleted 11y ago[deleted]
- k_bx 11y agoCurrently you can see everywhere ATMs are asking you to visually check that keypad and card-input looks exactly like on picture. With this technology, you don't need to make a fake keypad to "hear" PIN sequence, you could just "listen" to it from somewhere else, some place not seen well.
- melling 11y agoThere has been a lot of recent work with gesture based computing: Intel Real Sense, Google's Soli, Myo, Leap Motion https://github.com/melling/ErgonomicNotes/blob/master/README.org#gesture-computing https://github.com/melling/ErgonomicNotes/blob/master/README... Leap Motion made huge improvements a few weeks ago with their Orion SDK: http://venturebeat.com/2016/03/04/leap-motions-hyper-accurate-hand-tracking-vr-tech-shines-in-new-orion-demo/ http://venturebeat.com/2016/03/04/leap-motions-hyper-accurat... We must be close to actually getting something basic for our desktops.
- crudbug 11y agoSimilar to SixthSense [1] work from Media Lab. [1] https://www.media.mit.edu/research/highlights/sixthsense-wearable-gestural-interface-augment-our-world https://www.media.mit.edu/research/highlights/sixthsense-wea...
- verbatim 11y agoI can understand the "I", but what's the "O" part of this?
- strictnein 11y agoThe images drawn on screen?
- Eduard 11y agohm, that logic would mean a computer mouse is an input-output device, not only an input device.
- deleted 11y ago[deleted]
- dandare 11y agoI don't get it, how do they track specifically the tip of a finger? Or are they not?
- kennethkl 11y agothey probably track the closest object to the phone. just a guess. then ignore subsequent echos for a split second.
- leosteve78 11y agoIt's impressive!
- exotiik 11y agoFor a second i tought we were April 1st
- sehugg 11y agoI was thinking of something along these lines for a proximity sensor / motion detector application where you don't need very much accuracy.
- memonkey 11y agoWhat happens when there are multiple devices around each other emitting the signals? Great proof of concept.
- aub3bhat 11y agoThis is not same as FingerIO since it does not uses sophisticated signal processing but still interesting. Make sure that you remove earphones before using it. https://danielrapp.github.io/doppler/ https://danielrapp.github.io/doppler/
- nly 11y agoMy intuition tells me this just doesn't hold water with respect to information theory.. i.e. the number of bits of useful information about a finger you can pull from a microphone. Putting aside human digits, has anyone even demonstrated that you can reliably detect an eighteen-wheeler rig moving toward a phone with this technique? And what about the range of the speaker? Complete nonsense.
- et2o 11y agoAre you suggesting the video is fake? I think your intuition is wrong. To address your comment more directly, I don't see any information theory type limit immediately applicable here for finding xyz coordinates using echolocation. That's done in a variety of contexts.
- candu 11y agoYour intuition is probably wrong: off-the-shelf consumer-grade microphones can typically gather 44.1-96k samples per second at 16-24 bits per sample. That's a lot of potential information. Also, consider that your ear/brain apparatus estimates object positions and occlusions from audio signals all the time. Also also, the eighteen-wheeler problem is vastly different from the finger problem, as the latter is smaller, slower, and closer to the microphone, each by 1-2 orders of magnitude.
- rhema 11y agoFrom an information theory perspective, think about the number of samples. Even 22Khz yields a tremendous number of individual points.
- k__ 11y agoUltrasound tracking is always problematic, because of all the noise. I have the feeling, every few years someone has the idea again, to use ultrasound for something and it starts promising, but then the accuracy and lag doesn't go away and dogs and cats go wild.
- szczys 11y agoLatency looks like a real issue in this demo. If it can be improved this could be important, but think about how impatient you are if your smartphone doesn't respond to your touch immediately. Users have been trained to be irritated by laggy interfaces.
- parkaboy 11y agoRegarding everyone's latency concerns, as someone who has done low-latency audio processing on Android -- in their defense I'd bet almost anything the demo is meant to only demonstrate the math behind this. Depending on the platform (Android cough), low latency audio processing can be almost a dark art itself. And hey look, they're doing this on Android. My guess is that they decided to release the demo earlier instead of spending days/weeks getting up to speed with low-latency audio processing in the Android JNI. It's an academic demo/press release. Not a software release for production/market.
- jacobolus 11y agoA lot of production input devices for stuff like waving hand gestures and pen input also have unacceptably high latency. Getting latency down is hard, and makes a huge difference to usability.
- rsp1984 11y agoI'm curious, besides doing things in C/C++, are there any "magic tricks" to doing low-latency audio processing on Android? Looking at the chart here [1] it still seems to be a good bit behind iOS. [1] http://www.androidpolice.com/2015/11/13/android-audio-latency-in-depth-its-getting-better-especially-with-the-nexus-5x-and-6p/ http://www.androidpolice.com/2015/11/13/android-audio-latenc...
- parkaboy 11y agoIn the Java layer? About the only thing you can do is ensure that you're using the devices native sampling rate: typically 48 kHz for phones, and 44.1 kHz for tablets. Non-native sampling will induce a rather large latency hit. The buffering + buffer size stuff, unfortunately, is really only accessible in native IIRC. To be completely honest, it's been a long time since I've messed with audio-stuff on the Java-side, so not sure if/how much things have changed for the better.
- rsp1984 11y agoThanks. I actually meant in addition to doing the computation side of things in C/C++, are there any undocumented tricks or pitfalls to avoid?
- TTPrograms 11y agoHow many microphones do cell phones typically have? I guess I assumed one, though background noise cancelling would certainly be improved by having more. For this kind of positioning it seems the minimum needed would be 3 - and the Android SDK can access those audio streams separately unprocessed? Pretty neat.
- djsumdog 11y agoI have a feeling this is also really depending on the hardware. The demos were probably designed around the specific brand of watch and cellphone since they'd need to know exact distances between the microphones/speakers. It's a really cool concept. I wish they'd open source what they have, or at least have plans to open source it. However if this came about via University funding, they'll probably claim IP on it. If it was a student's own fellowship, he/she/they might decide to create a start-up out of it.
- eltronix 11y agoIt might as well be alchemy
- pizza 11y agosonar keylogger enabled