5 ms·

Tapster’s robots are built to poke touchscreens

64 points by hugs 9y ago

chrome4lyfe1233 9y ago
Very interesting.
jeffehobbs 9y ago
Ugggggggghhhgh. The repercussions of this, I don't want to think about.
- hugs 9y ago
  Tapster creator here. Which repercussions?
  - rhizome 9y ago
    The first thing I thought about was click-fraud.
    - hugs 9y ago
      Anything else?
      - jaclaz 9y ago
        If I may, there is a "niche", but most probably economically valid market in professional digital forensics and/or data recovery. The first (obvious) use is tapping access codes or swiping patterns to unlock a device (when/if the number of attempts is resettable [1]). The second is to document contents, on some phones (unlocked but for which there is no available imaging/copying solution or for specific apps for which there is no external reader), the current procedure is (say a communication app) to manually have a message on the screen, take a photo of it, swipe to the next one, take another photo, etc. See as an example/reference: https://www.forensicfocus.com/Forums/viewtopic/t=15977/ https://www.forensicfocus.com/Forums/viewtopic/t=15977/ [1] Since the reset in many cases is through a power reset, an accessory "push button tool" will be needed.
    - Buge 9y ago
      Click fraud on desktop is obviously much easier than on mobile. Even on mobile, ideally (for the fraudster) it would be done in an emulator. There are probably tricks that can be done to detect emulators though. I would consider those flaws in the emulator that should be fixed.
      - brownbat 9y ago
        Click farms are often a room full of phones on racks with cheap labor: https://www.zerohedge.com/news/2017-05-11/look-inside-chinese-click-farm-fake-followers-fake-likes-fake-reviews https://www.zerohedge.com/news/2017-05-11/look-inside-chines... I don't think they're primarily doing "click fraud" for ads so much as account creation (for spam), captcha defeats, and social media manipulation. Maybe that's all click (tap?) fraud though. Also... I'm not sure how Tapster compares to click farms on pricing. Aces them on legitimacy though. So... I don't think a Fortune 500 company is going to send their in development app over to Russian mobs for final bug testing. So they might be same thing but very very different market segments.
        Buge 9y ago
        I think the cat and mouse game of finding an emulator aberration vs fixing the emulator aberration is waited toward the finding side. Especially for complex systems like current mobile devices, and when they are often heavily locked down like iOS. At some point the effort to develop a good enough emulator is more expensive than just buying the physical devices. But another option beyond emulator vs physical devices is physical devices with physical tapping (Tapster) vs physical devices with emulated tapping. That last option seems to be what is used in your video. It might be too expensive to build an emulator, but it is cheaper to (jailbreak? and) emulate touch inputs than to pay a person or a robot to put in real touch inputs.
        brownbat 9y ago
        Ha, and now it hits me that I used the video without any actual humans... Maybe the cheap humans are falling by the wayside.
    - deleted 9y ago
      [deleted]
pimlottc 9y ago
Neat! I always wondered how hard it would be to build a machine to play those reaction-based ticket and prize redemption games.
- jjoonathan 9y ago
  Are they actually reaction-based or do they work like carnival / fair games where they're actually completely rigged and only pretend to involve reaction timing as a lure?
cocktailpeanuts 9y ago
Just out of curiosity, since the creator seems to be on this thread, I have a question: I'm sure there's an optimal "taps per second" number, where once you go over that threshold, the screens are unable to recognize the distinction. Can you share those numbers if you have one? Super interested.
- ChuckMcM 9y ago
  I'm not the original poster but the touch screen controller on a number of ST micro based kits I've played with 'debounces' the press to 10mS selectable down to 5mS. So figure slightly less than 100 to slightly less than 200 taps per second. That can also be affected by multitouch settings as well.
- hugs 9y ago
  (Tapster creator here) I've gotten my bots up to 800/taps per minute. I haven't seen / gotten to a higher upper limit, yet, where it is unable to recognize the distinction. However, what I have seen on iOS is after a few hundred taps, the taps stopped being recognized in that spot on the screen. Don't know if that's a bug or a feature of iOS to ignore touches at some point.
ChuckMcM 9y ago
Looks like a fun project, of course I wonder how much it would cost for 10 to 15 robots so that you could build your own little mobile ad click fraud farm.
- lnsru 9y ago
  I guess 100$ for Prusa Mendel 3D printer frame with steppers and controller and 100$ for some no-name tablet. Is this clicking thing profitable?
  - hugs 9y ago
    No, that's why most people use the robots instead for quality control / testing of the products they make.
    - lnsru 9y ago
      A company near by uses big industrial robots for credit card reader testing. I am just curious what grey applications are possible.
maxxxxx 9y ago
I always wanted to build a Lego robot to do UI automation on our Windows apps and embedded devices. Point a camera at the screen, recognize screen elements and then let the robot do the clicking.
- hugs 9y ago
  Do it!
pjc50 9y ago
These are rather fun. Our company has a more expensive oneoff solution, a traditional six-axis robot which we use for testing chip-and-pin integration by pressing the buttons on the pinpad.
bluesign 9y ago
In the context of automation (and click farms) if you are not testing the touch screen obviously, I think better approach can be by man in the middle attack with touchscreen and board. Or even emulate screen interface totally. I dont know if has anyone tried that before
- mschuster91 9y ago
  The point is, sometimes you do want to test the full environment including the touch screen. For example, you may want to check if the touchscreen locks up after a number of taps, if your OS/input parsing layer can cope with the jitter (a finger, in contrast to MITM-injected events, will almost never hit the same x/y coordinates in a row), or you want to check against real production hardware/software and not against something that's been modified for testing in any way.
- msgilligan 9y ago
  Actually, Samsung and Trustonic have a hardware-protected Trusted User Interface (TUI) that protects against man-in-the-middle attacks [1]. I work at Rivetz and we have developed software that displays in the TUI. Tapster looks like a great solution for automated testing for our environment. [1] https://www.trustonic.com/news/blog/benefits-trusted-user-interface/ https://www.trustonic.com/news/blog/benefits-trusted-user-in...

64 points by hugs 9y ago

chrome4lyfe1233 9y ago

Very interesting.

jeffehobbs 9y ago

Ugggggggghhhgh. The repercussions of this, I don't want to think about.

hugs 9y ago
Tapster creator here. Which repercussions?
- rhizome 9y ago
  The first thing I thought about was click-fraud.
  - hugs 9y ago
    Anything else?
    - jaclaz 9y ago
      If I may, there is a "niche", but most probably economically valid market in professional digital forensics and/or data recovery. The first (obvious) use is tapping access codes or swiping patterns to unlock a device (when/if the number of attempts is resettable [1]). The second is to document contents, on some phones (unlocked but for which there is no available imaging/copying solution or for specific apps for which there is no external reader), the current procedure is (say a communication app) to manually have a message on the screen, take a photo of it, swipe to the next one, take another photo, etc. See as an example/reference: https://www.forensicfocus.com/Forums/viewtopic/t=15977/ https://www.forensicfocus.com/Forums/viewtopic/t=15977/ [1] Since the reset in many cases is through a power reset, an accessory "push button tool" will be needed.
  - Buge 9y ago
    Click fraud on desktop is obviously much easier than on mobile. Even on mobile, ideally (for the fraudster) it would be done in an emulator. There are probably tricks that can be done to detect emulators though. I would consider those flaws in the emulator that should be fixed.
    - brownbat 9y ago
      Click farms are often a room full of phones on racks with cheap labor: https://www.zerohedge.com/news/2017-05-11/look-inside-chinese-click-farm-fake-followers-fake-likes-fake-reviews https://www.zerohedge.com/news/2017-05-11/look-inside-chines... I don't think they're primarily doing "click fraud" for ads so much as account creation (for spam), captcha defeats, and social media manipulation. Maybe that's all click (tap?) fraud though. Also... I'm not sure how Tapster compares to click farms on pricing. Aces them on legitimacy though. So... I don't think a Fortune 500 company is going to send their in development app over to Russian mobs for final bug testing. So they might be same thing but very very different market segments.
      - Buge 9y ago
        I think the cat and mouse game of finding an emulator aberration vs fixing the emulator aberration is waited toward the finding side. Especially for complex systems like current mobile devices, and when they are often heavily locked down like iOS. At some point the effort to develop a good enough emulator is more expensive than just buying the physical devices. But another option beyond emulator vs physical devices is physical devices with physical tapping (Tapster) vs physical devices with emulated tapping. That last option seems to be what is used in your video. It might be too expensive to build an emulator, but it is cheaper to (jailbreak? and) emulate touch inputs than to pay a person or a robot to put in real touch inputs.
        brownbat 9y ago
        Ha, and now it hits me that I used the video without any actual humans... Maybe the cheap humans are falling by the wayside.
  - deleted 9y ago
    [deleted]

pimlottc 9y ago

Neat! I always wondered how hard it would be to build a machine to play those reaction-based ticket and prize redemption games.

jjoonathan 9y ago
Are they actually reaction-based or do they work like carnival / fair games where they're actually completely rigged and only pretend to involve reaction timing as a lure?

cocktailpeanuts 9y ago

Just out of curiosity, since the creator seems to be on this thread, I have a question: I'm sure there's an optimal "taps per second" number, where once you go over that threshold, the screens are unable to recognize the distinction. Can you share those numbers if you have one? Super interested.

ChuckMcM 9y ago
I'm not the original poster but the touch screen controller on a number of ST micro based kits I've played with 'debounces' the press to 10mS selectable down to 5mS. So figure slightly less than 100 to slightly less than 200 taps per second. That can also be affected by multitouch settings as well.
hugs 9y ago
(Tapster creator here) I've gotten my bots up to 800/taps per minute. I haven't seen / gotten to a higher upper limit, yet, where it is unable to recognize the distinction. However, what I have seen on iOS is after a few hundred taps, the taps stopped being recognized in that spot on the screen. Don't know if that's a bug or a feature of iOS to ignore touches at some point.

ChuckMcM 9y ago

Looks like a fun project, of course I wonder how much it would cost for 10 to 15 robots so that you could build your own little mobile ad click fraud farm.

lnsru 9y ago
I guess 100$ for Prusa Mendel 3D printer frame with steppers and controller and 100$ for some no-name tablet. Is this clicking thing profitable?
- hugs 9y ago
  No, that's why most people use the robots instead for quality control / testing of the products they make.
  - lnsru 9y ago
    A company near by uses big industrial robots for credit card reader testing. I am just curious what grey applications are possible.

maxxxxx 9y ago

I always wanted to build a Lego robot to do UI automation on our Windows apps and embedded devices. Point a camera at the screen, recognize screen elements and then let the robot do the clicking.

hugs 9y ago
Do it!

pjc50 9y ago

These are rather fun. Our company has a more expensive oneoff solution, a traditional six-axis robot which we use for testing chip-and-pin integration by pressing the buttons on the pinpad.

bluesign 9y ago

In the context of automation (and click farms) if you are not testing the touch screen obviously, I think better approach can be by man in the middle attack with touchscreen and board. Or even emulate screen interface totally. I dont know if has anyone tried that before

mschuster91 9y ago
The point is, sometimes you do want to test the full environment including the touch screen. For example, you may want to check if the touchscreen locks up after a number of taps, if your OS/input parsing layer can cope with the jitter (a finger, in contrast to MITM-injected events, will almost never hit the same x/y coordinates in a row), or you want to check against real production hardware/software and not against something that's been modified for testing in any way.
msgilligan 9y ago
Actually, Samsung and Trustonic have a hardware-protected Trusted User Interface (TUI) that protects against man-in-the-middle attacks [1]. I work at Rivetz and we have developed software that displays in the TUI. Tapster looks like a great solution for automated testing for our environment. [1] https://www.trustonic.com/news/blog/benefits-trusted-user-interface/ https://www.trustonic.com/news/blog/benefits-trusted-user-in...