3 ms·

Another reason to use buttons: if you wanted to automate the button pushing, it's easier to create a computer vision function to look for round rectangles vs go

by hugs 3y ago

Another reason to use buttons: if you wanted to automate the button pushing, it's easier to create a computer vision function to look for round rectangles vs going full-blown machine learning trying to identify things that could be buttons. (Source: I make software and robots for pushing buttons.)

dewey 3y ago
Sounds like a reason why people would not use buttons as more often than not people actively have something against UI automation for mostly no good reason.
- dylan604 3y ago
  I'm assuming that if you're trying to automate with UI automation that there was no API provided? Automating UI interaction just seems like last resort for pro users and an act of desperation, or screen scrapers.
wrsh07 3y ago
For websites (eg UI testing) you normally annotate things to be clicked, so I don't find this super compelling Parsing text (eg html) is a bit easier than parsing images, although maybe the difference is going to zero
- hugs 3y ago
  Generally, I agree, if you can get access to the underlying structure/source, then you should use that. But I've been making UI test automation tools for 20 years. There are some cases where adding annotations is not possible or not feasible at low-cost, but the feature still needs to be automated.
- zogrodea 3y ago
  Fluent Search for Windows has a feature where typing a letter combination can click any element on the screen. I appreciate it for the same reason people like shortcuts. Fluent Search is a desktop app and this feature is meant to work regardless of which apps are on the screen (browser, Notepad, whatever else) so I think there's a legitimate point here although it might be a niche one not relevant to you. (I don't know enough about Windows internals to answer with confidence if a better solution is available like examining the OS accessibility tree, but I guess an advantage of Fluent Search's approach is that UIs from toolkits not implementing accessibility standards aren't excluded. The current approach has drawbacks like false positives too though because the ML model isn't 100% accurate.)
dylan604 3y ago
Why are robots needed for software button pressing?
- hugs 3y ago
  - Device provisioning - Automating a device right after power-on to do initial set-up, add to Wi-Fi, install apps, etc. (Other methods, like ADB, have to be enabled first. Who does the enabling?) - Security testing - Interacting with a device that may have malware on it. You use a robot so that it is 100% unmodified while you study it. For further safety, the device is run inside a Faraday box, and you use the robot to access the device from outside. (Otherwise, you do everything manually in a human-sized Faraday cage.) - Performance testing - Getting timing data of app performance is more accurate when the device is unmodified and you use an external tool to interact with the UI. - Testing medical devices - Required to be tested exactly the way a user would interact with the device. - Point of Sale testing - Some real world end-to-end tests involves inserting or tapping a card and entering a PIN on the screen. - Or you're interacting with a touchscreen that has no other programmable interface, but you still want to test automate it (like any touchscreen on any device that is not a phone or tablet, e.g. an HVAC or alarm system panel.) (These are all things I've been paid to make robots for over the last decade.)
- twobitshifter 3y ago
  example, I have to enter my information on a form every day but the form has no accessible api. I am just copying content from when place to another. Perfect use for a robot.
barelyauser 3y ago
If you're testing said buttons with a custom machine, don't you already know where the buttons are beforehand? Why would one need to analyze single frames and look for buttons using machine vision?

by hugs 3y ago

dewey 3y ago

Sounds like a reason why people would not use buttons as more often than not people actively have something against UI automation for mostly no good reason.

dylan604 3y ago
I'm assuming that if you're trying to automate with UI automation that there was no API provided? Automating UI interaction just seems like last resort for pro users and an act of desperation, or screen scrapers.

wrsh07 3y ago

For websites (eg UI testing) you normally annotate things to be clicked, so I don't find this super compelling Parsing text (eg html) is a bit easier than parsing images, although maybe the difference is going to zero

hugs 3y ago
Generally, I agree, if you can get access to the underlying structure/source, then you should use that. But I've been making UI test automation tools for 20 years. There are some cases where adding annotations is not possible or not feasible at low-cost, but the feature still needs to be automated.
zogrodea 3y ago
Fluent Search for Windows has a feature where typing a letter combination can click any element on the screen. I appreciate it for the same reason people like shortcuts. Fluent Search is a desktop app and this feature is meant to work regardless of which apps are on the screen (browser, Notepad, whatever else) so I think there's a legitimate point here although it might be a niche one not relevant to you. (I don't know enough about Windows internals to answer with confidence if a better solution is available like examining the OS accessibility tree, but I guess an advantage of Fluent Search's approach is that UIs from toolkits not implementing accessibility standards aren't excluded. The current approach has drawbacks like false positives too though because the ML model isn't 100% accurate.)

dylan604 3y ago

Why are robots needed for software button pressing?

hugs 3y ago
- Device provisioning - Automating a device right after power-on to do initial set-up, add to Wi-Fi, install apps, etc. (Other methods, like ADB, have to be enabled first. Who does the enabling?) - Security testing - Interacting with a device that may have malware on it. You use a robot so that it is 100% unmodified while you study it. For further safety, the device is run inside a Faraday box, and you use the robot to access the device from outside. (Otherwise, you do everything manually in a human-sized Faraday cage.) - Performance testing - Getting timing data of app performance is more accurate when the device is unmodified and you use an external tool to interact with the UI. - Testing medical devices - Required to be tested exactly the way a user would interact with the device. - Point of Sale testing - Some real world end-to-end tests involves inserting or tapping a card and entering a PIN on the screen. - Or you're interacting with a touchscreen that has no other programmable interface, but you still want to test automate it (like any touchscreen on any device that is not a phone or tablet, e.g. an HVAC or alarm system panel.) (These are all things I've been paid to make robots for over the last decade.)
twobitshifter 3y ago
example, I have to enter my information on a form every day but the form has no accessible api. I am just copying content from when place to another. Perfect use for a robot.

barelyauser 3y ago

If you're testing said buttons with a custom machine, don't you already know where the buttons are beforehand? Why would one need to analyze single frames and look for buttons using machine vision?