3 ms·

Isn't "computer use" just interaction with a shell-like environment, which is routine for current agents?

vineyardmike 9mo ago
No. Computer use (to anthropic, as in the article) is an LLM controlling a computer via a video feed of the display, and controlling it with the mouse and keyboard.
- cowboylowrez 9mo ago
  oh hell no haha maybe with THEIR login hahaha
- dbbk 9mo ago
  That sounds weird. Why does it need a video feed? The computer can already generate an accessibility tree, same as Playwright uses it for webpages.
  - 0sdi 9mo ago
    So that it can utilize gui and interfaces designed for humans. Think of video editing program for example.
    - dbbk 9mo ago
      Yes. GUIs expose an accessibility tree.
      - slopinthebag 9mo ago
        Not all of them do, and not all of the ones that do expose enough to be useful to the AI.
      - bowsamic 9mo ago
        Even if they do (often not the case) this will be far from exhaustive, and likely won’t reflect the structure of the application very well. Vision based testing is often combined with accessibility based testing
  - lsaferite 9mo ago
    I feel like a legion of blind computer users could attest to how bad accessibility is online. If you added AI Agents to the users of accessibility features you might even see a purposeful regression in the space.
- chasd00 9mo ago
  > controlling a computer via a video feed of the display, and controlling it with the mouse and keyboard. I guess that's one way to get around robots.txt. Claim that you would respect it but since the bot is not technically a crawler it doesn't apply. It's also an easier sell to not identify the bot in the user agent string because, hey, it's not a script, it's using the computer like a human would!
- jebus989 9mo ago
  Even simpler it just takes screenshots (or at least that's what it was doing last time I used it)
zmmmmm 9mo ago
No their definition of "computer use" now means: > where the model interacts with the GUI (graphical userinterface) directly.
michaelt 9mo ago
> Almost every organization has software it can’t easily automate: specialized systems and tools built before modern interfaces like APIs existed. [...] > hundreds of tasks across real software (Chrome, LibreOffice, VS Code, and more) running on a simulated computer. There are no special APIs or purpose-built connectors; the model sees the computer and interacts with it in much the same way a person would: clicking a (virtual) mouse and typing on a (virtual) keyboard. https://www.anthropic.com/news/claude-sonnet-4-6 https://www.anthropic.com/news/claude-sonnet-4-6
jpalepu 9mo ago
Interesting question! In this context, "computer use" means the model is manipulating a full graphical interface, using a virtual mouse and keyboard to interact with applications (like Chrome or LibreOffice), rather than simply operating in a shell environment.
- mentalgear 9mo ago
  Indeed GUI-use would have been the better naming.
lukev 9mo ago
This is being downvoted but it shouldn't be. If the ultimate goal is having a LLM control a computer, round-tripping through a UX designed for bipedal bags of meat with weird jelly-filled optical sensors is wildly inefficient. Just stay in the computer! You're already there! Vision-driven computer use is a dead end.
- chasd00 9mo ago
  i replied as much to a sibling comment but i think this is a way to wiggle out of robots.txt, identifying user agent strings, and other traditional ways for sites to filter for a bot.
  - mvdtnz 9mo ago
    I mean if they want to "wriggle out" of robots.txt they can just ignore it. It's entirely voluntary.
  - lukev 9mo ago
    Right but those things exist to prevent bots. Which this is. So at this point we're talking about participating in the (very old) arms race between scrapers & content providers. If enough people want agents, then services should (or will) provide agent-compatible APIs. The video round-trip remains stupid from a whole-system perspective.
- ashirviskas 9mo ago
  Someone ping me in 5 years, I want to see if this aged like milk or wine
  - JSR_FDED 9mo ago
    “Computer, respond to this guy in 5 years”
- zmmmmm 9mo ago
  you could say that about natural language as well, but it seems like having computers learn to interface with natural language at scale is easier than teaching humans to interface using computer languages at scale. Even most qualified people who work as software programmers produce such buggy piles of garbage we need entire software methodologies and testing frameworks to deal with how bad it is. It won't surprise me if visual computer use follows a similar pattern. we are so bad at describing what we want the computer to do that it's easier if it just looks at the screen and figures it out.

No. Computer use (to anthropic, as in the article) is an LLM controlling a computer via a video feed of the display, and controlling it with the mouse and keyboard.

cowboylowrez 9mo ago
oh hell no haha maybe with THEIR login hahaha
dbbk 9mo ago
That sounds weird. Why does it need a video feed? The computer can already generate an accessibility tree, same as Playwright uses it for webpages.
- 0sdi 9mo ago
  So that it can utilize gui and interfaces designed for humans. Think of video editing program for example.
  - dbbk 9mo ago
    Yes. GUIs expose an accessibility tree.
    - slopinthebag 9mo ago
      Not all of them do, and not all of the ones that do expose enough to be useful to the AI.
    - bowsamic 9mo ago
      Even if they do (often not the case) this will be far from exhaustive, and likely won’t reflect the structure of the application very well. Vision based testing is often combined with accessibility based testing
- lsaferite 9mo ago
  I feel like a legion of blind computer users could attest to how bad accessibility is online. If you added AI Agents to the users of accessibility features you might even see a purposeful regression in the space.
chasd00 9mo ago
> controlling a computer via a video feed of the display, and controlling it with the mouse and keyboard. I guess that's one way to get around robots.txt. Claim that you would respect it but since the bot is not technically a crawler it doesn't apply. It's also an easier sell to not identify the bot in the user agent string because, hey, it's not a script, it's using the computer like a human would!
jebus989 9mo ago
Even simpler it just takes screenshots (or at least that's what it was doing last time I used it)

zmmmmm 9mo ago

No their definition of "computer use" now means: > where the model interacts with the GUI (graphical userinterface) directly.

michaelt 9mo ago

> Almost every organization has software it can’t easily automate: specialized systems and tools built before modern interfaces like APIs existed. [...] > hundreds of tasks across real software (Chrome, LibreOffice, VS Code, and more) running on a simulated computer. There are no special APIs or purpose-built connectors; the model sees the computer and interacts with it in much the same way a person would: clicking a (virtual) mouse and typing on a (virtual) keyboard. https://www.anthropic.com/news/claude-sonnet-4-6 https://www.anthropic.com/news/claude-sonnet-4-6

jpalepu 9mo ago

Interesting question! In this context, "computer use" means the model is manipulating a full graphical interface, using a virtual mouse and keyboard to interact with applications (like Chrome or LibreOffice), rather than simply operating in a shell environment.

mentalgear 9mo ago
Indeed GUI-use would have been the better naming.

lukev 9mo ago

This is being downvoted but it shouldn't be. If the ultimate goal is having a LLM control a computer, round-tripping through a UX designed for bipedal bags of meat with weird jelly-filled optical sensors is wildly inefficient. Just stay in the computer! You're already there! Vision-driven computer use is a dead end.

chasd00 9mo ago
i replied as much to a sibling comment but i think this is a way to wiggle out of robots.txt, identifying user agent strings, and other traditional ways for sites to filter for a bot.
- mvdtnz 9mo ago
  I mean if they want to "wriggle out" of robots.txt they can just ignore it. It's entirely voluntary.
- lukev 9mo ago
  Right but those things exist to prevent bots. Which this is. So at this point we're talking about participating in the (very old) arms race between scrapers & content providers. If enough people want agents, then services should (or will) provide agent-compatible APIs. The video round-trip remains stupid from a whole-system perspective.
ashirviskas 9mo ago
Someone ping me in 5 years, I want to see if this aged like milk or wine
- JSR_FDED 9mo ago
  “Computer, respond to this guy in 5 years”
zmmmmm 9mo ago
you could say that about natural language as well, but it seems like having computers learn to interface with natural language at scale is easier than teaching humans to interface using computer languages at scale. Even most qualified people who work as software programmers produce such buggy piles of garbage we need entire software methodologies and testing frameworks to deal with how bad it is. It won't surprise me if visual computer use follows a similar pattern. we are so bad at describing what we want the computer to do that it's easier if it just looks at the screen and figures it out.