7 ms·
Keylogging on IPhone and Android Using Gyroscope Data and Machine Learning
- jbob2000 9y agoTL;DR it's possible to detect where you tapped on your phone screen somewhat reliably. Theoretically, someone could develop an app to associate screen location taps to keys and could pick out a password. Authors suggest phone manufacturers should limit access to accelerometers to prevent the attack. This is extremely contrived and would take a shitload of skilled work to get right. It's way easier to make a phishing page coupled with social engineering to get what you want.
- anfractuosity 9y agoWould you be able to access screen taps from another app in the background?
- yorwba 9y agoI think step counters depend on being able to read accelerometer data. You might be able to add enough noise to the signal to make tap localization impossible while still keeping steps detectable, but that requires careful tuning.
- djrogers 9y agoOn iOS at least, most ‘step counter’ apps don’t acually do the step counting - they just pull the data from HealthKit, as the phone is already tracking them.
- skoocda 9y agoYou're only accessing accelerometer / gyro data here, so yes this could occur in the background. On iPhone it restricts background activity after 10 minutes, but on Android you'd have free reign until they manually close the app.
- anfractuosity 9y agoWait sorry, I thought the person I replied to was referring to accessing screen tap x,y coords from the OS. Sorry, looks like I misunderstood them (didn't notice they were referring to the paper).
- skoocda 9y agoOh my mistake, it was actually I who misread the first comment. Regardless, I don't think a screen tap logger would be necessary in many cases, since you could just open the keyboard on the phone and manually check the bounding boxes for each key. Making this work on N phones would be more difficult, but not unfeasible. You'd probably just need to know a few things about the phone (resolution, screen size), and it wouldn't be hard to find that information. Would be especially easy to just target the limited iPhone line.
- moheeb 9y agoIs this why most password entry forms in Android do not allow use of the Swype-type keyboard? I always thought that was strange. Now it makes more sense....backdooring
- Unknoob 9y agoI was a bit disappointed by the lack of a real scenario implementation. While the data suggests it might work, I wanted to see how precise it would be in a simple implementation like the author described towards the end, where a simple game of tapping trains the NN and then uses the information to try and capture a typed password. If I got that close to making something as cool as this, I would not stop experimenting until I found out whether it worked or not. That is, unless I tested it, found that the imprecisions made it impossible to obtain reliable data, and decided to not write about it to not discredit all the work I had done. I really hope someone takes the idea and tries it out to assess how viable it is, then I can be paranoid about 2 more sensors spying on me.
- andrewguenther 9y agoThe Usenix paper they referenced from 2011 did exactly this. https://www.usenix.org/legacy/event/hotsec11/tech/final_files/Cai.pdf https://www.usenix.org/legacy/event/hotsec11/tech/final_file... They focused purely on the number keyboard to try and extract PIN numbers and were able to achieve 70% accuracy using only a basic classifier.
- Unknoob 9y agoThank you for reminding me of the references, I had forgotten to check them out! I wonder how much newer smartphones with bigger screens and more precise sensors could improve these results. It's scary to think of how many different sensors could be used to gather even more information to make it more precise.
- anfractuosity 9y agoVery cool! Also this is mindblowing, logging keystrokes from a standard keyboard using wifi. "we show for the first time that WiFi signals can also be exploited to recognize keystrokes. The intuition is that while typing a certain key, the hands and fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time-series of Channel State Information (CSI) values" https://www.sigmobile.org/mobicom/2015/papers/p90-aliA.pdf https://www.sigmobile.org/mobicom/2015/papers/p90-aliA.pdf Edit: I wonder as an aside how precisely you could potentially use the accelerometer etc. for dead-reckoning.
- gregorymfoster 9y agoCo-author here :) In response to dead-reckoning: https://medium.com/@gregorymfoster/how-any-app-could-track-the-indoor-location-of-everyone-magnetic-localization-acf3707716de https://medium.com/@gregorymfoster/how-any-app-could-track-t...
- anfractuosity 9y agoAwesome thanks a lot, I'd not seen that! It looks very in-depth (which is what I wanted :), I'll have to have a good look through it. Edit: Wow, I'm just skimming it at the moment, I'd never heard of 'Magnetic Particle Filtering' before, that is really amazing! Also semi-related, I just found this https://en.wikipedia.org/wiki/Magnetic_anomaly_detector https://en.wikipedia.org/wiki/Magnetic_anomaly_detector. Apparently they degauss submarines precisely to avoid that :)
- gregorymfoster 9y agoThanks!
- jclardy 9y agoWouldn't the model be different for each user? and different for different typing scenarios? Like typing one handed vs two, sitting on a bus, in a car, while walking, versus just sitting on the couch. It seems like it could be possible, but it would require a lot of training. Get it into a popular note-taking app to record keystroke + accelerometer data. Also a simple fix for iOS as a platform would just be blocking out or filtering motion data when the secure keyboard is showing. I assume a similar thing could be done on android.
- Etheryte 9y agoThey address the case of different users and differing inputs in the "Unified Pipeline" section.
- ninju 9y agoTo allow for training of the ML to differences of each users the authors describe have the end-user first play a game Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on the screen mindlessly to advance some objective. During this tapping, the app uses transfer learning to tailor the model to the user and test its own predictive capacity. Very sneaky indeed...and I loved Evil Flappy :-)
- sundaysailor 9y agoAttacks like these can be easily dwarfed by employing a nonstandard virtual keyboard which displays a randomly permutated key layout during PIN entry. Some online banking web apps did this already ten years ago as a defense against "mouseloggers". Using a good RNG is advised though.
- bllguo 9y agoCome to think of it, I can't remember the last time I typed a password in an app, after initial setup.
- tinus_hn 9y agoQuite surprising iOS allows that kind of sample rate for a background app.
- willstrafach 9y agoI believe in iOS 8 and up (could even be earlier) Apple closed up tricks which would have allowed this sort of thing to work.
- thinkMOAR 9y agoNeat, though i'd call it key-estimation rather then logging, as i consider logging a very accurate, 1:1 log of what actually happened, this could have some errors in it, not making it less 'dangerous', lack of better word. But for now i'll have to randomly rotate my device at each input... And or get one of those keyboards that shuffles the letters around the keyboard at each input.