3 ms·

As a disclaimer, my name is George (https://george.nychis.com https://george.nychis.com) and I helped found a software automation company based out of Boston, M

As a disclaimer, my name is George (https://george.nychis.com https://george.nychis.com) and I helped found a software automation company based out of Boston, MA that is "related" to this space called Soroco (https://www.soroco.com https://www.soroco.com). Our focus is around solving the kinds of problems you talk about. Building automation to replace repetitive and deterministic work across software systems in the enterprise. As you have learned, RPA is mostly screen scraping, macros, and as another person referred to it below "Rube Goldberg machines" that break when UIs change. For all of these reasons above (the need for automation and the limitations of RPA), we started Soroco. Like you said (and we agree) you can't build serious automation without real development. We take a fundamentally different approach which is to build a platform/SDK for automation that provides a significant amount of reliability on top of a number of unreliable automation layers. There is no other way to automate Windows applications or Java applications than through either A) Screen scraping, or B) Their accessibility layers (e.g., Windows UIA). The problem is that BOTH are extremely unreliable. For web, that equivalent is Selenium (at least as the primary layer). As another commenter mentioned below, a "watered down version of Selenium" is exactly what is NOT needed -- nor are Windows UIA or other accessibility layers great. They are all extremely unreliable. When something like Selenium fails to find an element, what do you do? Halt your business critical process? All of these frameworks were never built to handle business critical processes, they were mainly built for application testing. The RPA industry has basically tried to leverage them and pretend they are enterprise ready by hiding them behind flow chart builders. They are NOT enterprise ready. To draw an analogy, it's like having the Internet and IP, which do NOT provide or guarantee reliability. You need the equivalent of TCP and reliability layers. At Soroco, we have dealt with all of this and have put a tremendous amount of effort into building reliability layers on these unreliable systems, along with security, scalability, and machine learning. We construct automation systems using a full programming language (Python) and have built flow control layers and reliability on top of it. Sort of like an automation SDK for Python with a full IDE. We have lots of supporting microservices for storing credentials securely, storing information scalably, and deploying. We've even open sourced how we encrypt Python (https://blog.soroco.com/ https://blog.soroco.com/) What does "cognitive" mean in this field? At Soroco, we don't pretend automation systems can "self heal" or "self construct" -- that's just silly. For Soroco, cognitive means that if you had your automation system do the same thing over and over (e.g., during User Acceptance Testing)... what can it learn about the distribution of inputs or outputs to flag anomalous behavior? We also apply Machine Learning and Computer Vision to specific use cases to make them achieve what is not possible through a laundry list of conditionals. For example, ML/NLP to classify intents of text and to do things like binary or multi-class classification. There's a lot of noise in the RPA space. I'd be happy to share more with you if you want to reach out directly to me: george@soroco.com or gnychis@gmail.com

phaedrus 8y ago
Are you familiar with Automa? I tried to get my organization to purchase it for automated testing, but approval for that ran aground on the fact that Automa doesn't have a US distributor. (As you may infer, I'm a gov't contractor.) I was hoping that your company offerred a product similar to Automa, but it looks like your business model is different to that. Anyway I ended up having to re-implement my Automa-using Python scripts using a different technology (TestStack.White and C#) and I 100% concur with your analogy about TCP over IP and building on unreliable foundation. It's also analogous to the sensor fusion problem of AI: picking out what to do when different methods of interacting with the automation layer report conflicting results. (And you have multiple ways because you need the redundancy to detect and deal with problem #1 - automation interfaces are incredibly unreliable.) For example between IT-mandated invasive antivirus software injecting shims in the OS and/or screwy win32 code in the program-under-test, or bugs in TestStack.White, (I've found indications of all three causes) there is some kind of randomness or ambiguity in the Window handles you get when you ask for certain controls on a dialog. You might ask for the OK button and get the button 50% of the time and 50% of the time get a handle which turns out to be the textbox above it. So I've taken a belt-and-suspenders approach and built a database of all of the controls I expect to find and their screen positions, a folder full of screenshot snippets which I intend to make my test automation program compare, and use that to augment what Windows UIA is telling me is on-screen. Due to the problem of receiving the wrong window handle (on this app) when I ask TestStack.White to find a control at a certain position or by name, I resort to just cross-collating the entire list of controls on the window with the best match from the database. However even this is 4x harder than it should be, because for some reason the positions and sizes in the database (obtained by a previous attempt with a different automation tool) differ by small pixel amounts from what TestStack.White reports. So I had to implement a fuzzy match. And then the entire object-oriented architecture of my automation utility program collapsed on itself like a house of cards, because now I have 2 or 3 different derived classes to represent "the same" conceptual object: if you code "button.Click()" should it click the button with the handle found by TestStack.White? Or click the coordinate from the database? Or click where the screenshot matched? Or do nothing because this is a mock object? Worse, I now have a constellation of 1..N OOP instances associated with each conceptual object: maybe the database says I should have a button, but I can't find a handle for it thus I cannot create a ButtonFromHandle but no matter I can still click on the center coord of ButtonFromData. But if I have both I really want to default to one (which?) that is most reliable. Object-oriented programming design methodologies are completely inadequate to deal with the case of "some number between 1 and N instances actually represent one conceptual object and should be treated like a sheaf of overlaid transparencies". Yes, it can be done, but the design turns into something that makes sense from that aspect but makes no sense from the API-user's point of view: you just want to click "the" button you don't want to do "button.Implementations.First().Click()" or some such. Then add in multiple layers of mocks for unit testing the automation tool and you get N*M combinations... Not sure where I was going with this, I guess I'm just excited to find someone who's worked on the same (or similar) problem.

phaedrus 8y ago

Are you familiar with Automa? I tried to get my organization to purchase it for automated testing, but approval for that ran aground on the fact that Automa doesn't have a US distributor. (As you may infer, I'm a gov't contractor.) I was hoping that your company offerred a product similar to Automa, but it looks like your business model is different to that. Anyway I ended up having to re-implement my Automa-using Python scripts using a different technology (TestStack.White and C#) and I 100% concur with your analogy about TCP over IP and building on unreliable foundation. It's also analogous to the sensor fusion problem of AI: picking out what to do when different methods of interacting with the automation layer report conflicting results. (And you have multiple ways because you need the redundancy to detect and deal with problem #1 - automation interfaces are incredibly unreliable.) For example between IT-mandated invasive antivirus software injecting shims in the OS and/or screwy win32 code in the program-under-test, or bugs in TestStack.White, (I've found indications of all three causes) there is some kind of randomness or ambiguity in the Window handles you get when you ask for certain controls on a dialog. You might ask for the OK button and get the button 50% of the time and 50% of the time get a handle which turns out to be the textbox above it. So I've taken a belt-and-suspenders approach and built a database of all of the controls I expect to find and their screen positions, a folder full of screenshot snippets which I intend to make my test automation program compare, and use that to augment what Windows UIA is telling me is on-screen. Due to the problem of receiving the wrong window handle (on this app) when I ask TestStack.White to find a control at a certain position or by name, I resort to just cross-collating the entire list of controls on the window with the best match from the database. However even this is 4x harder than it should be, because for some reason the positions and sizes in the database (obtained by a previous attempt with a different automation tool) differ by small pixel amounts from what TestStack.White reports. So I had to implement a fuzzy match. And then the entire object-oriented architecture of my automation utility program collapsed on itself like a house of cards, because now I have 2 or 3 different derived classes to represent "the same" conceptual object: if you code "button.Click()" should it click the button with the handle found by TestStack.White? Or click the coordinate from the database? Or click where the screenshot matched? Or do nothing because this is a mock object? Worse, I now have a constellation of 1..N OOP instances associated with each conceptual object: maybe the database says I should have a button, but I can't find a handle for it thus I cannot create a ButtonFromHandle but no matter I can still click on the center coord of ButtonFromData. But if I have both I really want to default to one (which?) that is most reliable. Object-oriented programming design methodologies are completely inadequate to deal with the case of "some number between 1 and N instances actually represent one conceptual object and should be treated like a sheaf of overlaid transparencies". Yes, it can be done, but the design turns into something that makes sense from that aspect but makes no sense from the API-user's point of view: you just want to click "the" button you don't want to do "button.Implementations.First().Click()" or some such. Then add in multiple layers of mocks for unit testing the automation tool and you get N*M combinations... Not sure where I was going with this, I guess I'm just excited to find someone who's worked on the same (or similar) problem.