At https://openadapt.ai/ https://openadapt.ai/ we are using LLMs to automate repetitive tasks in GUI interfaces. Think robotic process automation, but via learn

At https://openadapt.ai/ https://openadapt.ai/ we are using LLMs to automate repetitive tasks in GUI interfaces. Think robotic process automation, but via learning from demonstration rather than no-code scripting. The stack is mostly python running locally, and calling the OpenAI API (although we have plans to support offline models). For better visual understanding, we use a custom fork of Set-of-Mark prompting (https://github.com/microsoft/SoM https://github.com/microsoft/SoM) deployed to EC2 (see https://github.com/OpenAdaptAI/SoM/pull/3 https://github.com/OpenAdaptAI/SoM/pull/3).