4 ms·

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those tools is to act as a middle man to programmatically interpret and interact with what's on screen.

danielbln 2y ago
I think the reason is that this is the most general implementation. It doesn't need playwright or have access to the DOM or anything else, if it has a screen and mouse/keyboard, then it will work. That's quite powerful (if slow and pricey, at the moment).
- _heimdall 2y ago
  Unless I'm mistaken, playwright doesn't actually use the accessibility tree directly. It does have quite a few APIs for accessing nodes based on a11y attributes, but I could have sworn those were glorified query selectors rather than directly accessing the accessibility tree. Last I checked on it, maybe a year ago, there were browser proposals for standardizing the accessibility tree APIs but they were very early discussions and seemed pretty well stuck. That would be a good reason for Anthropic using image processing here though, short of forking open source a11y tools there may not have been a simple way to use accessibility data to interact.
infecto 2y ago
Those sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.
- ryukafalz 2y ago
  Accessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.
  - infecto 2y ago
    Accessibility tools are I find are never as great as the source. Just because they are made for humans does not mean they are an improvement. I imagine at best a stopgap as image models improve.
  - lupusreal 2y ago
    It's like making a robot that can walk up stairs instead of roll up wheelchair ramps or use elevators. It's harder, but more capable.
elif 2y ago
Crazy that this needs to be said but 'Computer use' is far more expansive of a domain than Internet browsing...
- _heimdall 2y ago
  From what I've seen of this new product (I've never used it), it sounds like its specifically trying to mimic a human user and they went with image recognition plus faked input devices. That approach is a weird one to me, though only as long as its limited to the current use. If this is just another test bed for a much more broad tool that could rely on accessibility APIs that makes sense.
ctoth 2y ago
Hmmm. I think I'm a blind user of the late 2024 Internet. I use a screen reader for everything. Now I'm paranoid that I'm just a computer use model, testing this a11y API hypothesis, in training.

danielbln 2y ago

I think the reason is that this is the most general implementation. It doesn't need playwright or have access to the DOM or anything else, if it has a screen and mouse/keyboard, then it will work. That's quite powerful (if slow and pricey, at the moment).

_heimdall 2y ago
Unless I'm mistaken, playwright doesn't actually use the accessibility tree directly. It does have quite a few APIs for accessing nodes based on a11y attributes, but I could have sworn those were glorified query selectors rather than directly accessing the accessibility tree. Last I checked on it, maybe a year ago, there were browser proposals for standardizing the accessibility tree APIs but they were very early discussions and seemed pretty well stuck. That would be a good reason for Anthropic using image processing here though, short of forking open source a11y tools there may not have been a simple way to use accessibility data to interact.

infecto 2y ago

Those sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.

ryukafalz 2y ago
Accessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.
- infecto 2y ago
  Accessibility tools are I find are never as great as the source. Just because they are made for humans does not mean they are an improvement. I imagine at best a stopgap as image models improve.
- lupusreal 2y ago
  It's like making a robot that can walk up stairs instead of roll up wheelchair ramps or use elevators. It's harder, but more capable.

elif 2y ago

Crazy that this needs to be said but 'Computer use' is far more expansive of a domain than Internet browsing...

_heimdall 2y ago
From what I've seen of this new product (I've never used it), it sounds like its specifically trying to mimic a human user and they went with image recognition plus faked input devices. That approach is a weird one to me, though only as long as its limited to the current use. If this is just another test bed for a much more broad tool that could rely on accessibility APIs that makes sense.

ctoth 2y ago

Hmmm. I think I'm a blind user of the late 2024 Internet. I use a screen reader for everything. Now I'm paranoid that I'm just a computer use model, testing this a11y API hypothesis, in training.