3 ms·

it’s a good q - we experimented a lot with computer use / agentic automation and found that at scale a hybrid solution where the automations run as deterministi

it’s a good q - we experimented a lot with computer use / agentic automation and found that at scale a hybrid solution where the automations run as deterministic code with agents for recovery is the best - running automations as code is faster & cheaper & when you’re doing critical tasks (like updating patient records) you don’t want an agent to potentially mess something up. previously writing RPA code used to take a long time - using AI (and its infinite patience) we can write more durable code that covers more edge cases And since they’re code based it’s pretty straightforward to an agents monitor them and update their code when upgrades to the underlying system happen etc… for observability - we have workflow execution logs that store text, videos and screenshots so an agent or a human can debug them - lots and lots of webhooks when things break ! (:

_crowecawcaw 4mo ago
I also experimented with vision/screenshot based computer use tools for similar use cases but had inconsistent results. LLMs had trouble getting precise pixel coordinates from a screenshot to move a mouse. And the screenshots took extra tokens. I had a lot more success using accessibility APIs to replace screenshots + input simulation since accessibility data is easier for LLMs to process. The accessibility functionality is now released as a separate library for building automation tooling: https://xa11y.dev/ https://xa11y.dev/
- fchishtie 4mo ago
  cool! thank you for sharing - will check it out
a-dub 4mo ago
that sounds like the way. also keeps runtime costs down. seems like the trick there would be to build (a|on top of) durable rpa librar(y|ies) that the agents and humans work in so that the automation recipes and their automated updates can be quickly skimmed and sanity checked when needed. add in some live automated testing (assuming you can make this happen with the legacy systems) and maybe you could get really close to fully automating all of it.

_crowecawcaw 4mo ago

I also experimented with vision/screenshot based computer use tools for similar use cases but had inconsistent results. LLMs had trouble getting precise pixel coordinates from a screenshot to move a mouse. And the screenshots took extra tokens. I had a lot more success using accessibility APIs to replace screenshots + input simulation since accessibility data is easier for LLMs to process. The accessibility functionality is now released as a separate library for building automation tooling: https://xa11y.dev/ https://xa11y.dev/

fchishtie 4mo ago
cool! thank you for sharing - will check it out

a-dub 4mo ago

that sounds like the way. also keeps runtime costs down. seems like the trick there would be to build (a|on top of) durable rpa librar(y|ies) that the agents and humans work in so that the automation recipes and their automated updates can be quickly skimmed and sanity checked when needed. add in some live automated testing (assuming you can make this happen with the legacy systems) and maybe you could get really close to fully automating all of it.