3 ms·
I’m curious how this looks in reality. They install a tiny client on their machine that follows their clicks, evaluates what’s on the screen and then does more
by Aeolun 6y ago
I’m curious how this looks in reality. They install a tiny client on their machine that follows their clicks, evaluates what’s on the screen and then does more clicks?
Then once it’s made the first time, it runs on their machine again. Or it runs on some central server?
- isp 6y agoThink about it as a programming language but with integrated support for automating user interfaces. Typical workflow for an "unattended" automation (which runs on a headless VM in a data centre somewhere) would be: - Run "Developer Studio" on a dev machine - Write "code" (in a visual programming language) to tell the software how to interact with other user interfaces (there are various different methods for this: via browser plugin to interact with the DOM in a web browser, or using screenreader APIs for native Windows applications, or even via OCR-like methods if all else fails) - Save the automation "source code" (it's a visual programming language, based on VB.NET/C#, and the underlying source code is XAML files), ideally add to version control - "Publish" the new automation to a central "Orchestrator" server - which is responsible for pushing the new automation code to other headless VMs, and coordinating execution
- ethbr0 6y agoTo add one addition piece: selectors / match rules. The key to getting automation reliable is being able to reliably identify target objects (e.g. button, grid cell, etc). ... reliably identify it, regardless of what screwed up, unsupported, what-were-these-devs-thinking underlying code. I once had to automate a webpage that was dynamically built, at user runtime, from a mainframe screen. Of course without any classes or id tags anywhere. So if the tool doesn't give you a toolbox big enough to handle pathological cases, it's not really fit for purpose.
- donarb 6y agoIt's not always just a screen logging thing. You can automate manual non-computer stuff as well. For example I had a client who had a manual process where a clerk entered info from physical cancelled checks into a spreadsheet. The spreadsheet was then fed into their accounting system for balancing. I wrote some Python code that would take a PDF of scanned check images, run the file through Google vision to OCR it, then pull the check info from the scanned data and put it into the spreadsheet. The accuracy is pretty good and any errors are put into a comment column of the spreadsheet where the user can review it. This code runs in a Windows server and becomes a UIPath bot in the system. So the user who is not very technically inclined merely has to log into the system, upload a scanned PDF and will receive the scanned results after the batch runs.