3 ms·

Am I the only one thinking this is an awful way for AI to do useful stuff for you? Why would I train an AI to use a GUI? Wouldn’t it be better to just have the

Am I the only one thinking this is an awful way for AI to do useful stuff for you? Why would I train an AI to use a GUI? Wouldn’t it be better to just have the AI learn API docs and use that? I don’t want the AI to open my browser, open google maps and search for Shawarma, I want the AI to call a google api and give me the result.

voiper1 2y ago
Sure, it's more effecient to have it use an API. And people have been integrating those for the last while. But there's tons of applications that are locked behind a website or deckstop GUI with no API that are accessible via vision.
famouswaffles 2y ago
The vast majority of Applications cannot be used by anything other than a GUI. We built computers to be used by humans and humans overwhelmingly operate computers with GUIs. So if you want a machine that can potentially operate computers as well as humans then you're going to have to stick to GUIs. It's the same reason we're trying to build general purpose robots in a human form factor. The fact that a car is about as wide as a two horse drawn carriage is also no coincidence. You can't ignore existing infrastructure.
- echoangle 2y ago
  But I don’t want an AI to „operate a computer“… maybe I’m missing the point of this but I just can’t imagine a usecase where this is a good solution. For everything browser based, the burden of making an API is probably relatively small and if the page is simple enough, you could maybe even get away with training the AI on the page HTML and generating a response to send. And for everything that’s not browser-based, I would either want the AI embedded in the software (image editors, IDEs…) or not there are all.
  - famouswaffles 2y ago
    >But I don’t want an AI to „operate a computer“… You don't and that's fine but certainly many people are interested in such a thing. >maybe I’m missing the point of this but I just can’t imagine a usecase where this is a good solution. If it could operate computers robustly and reliabily then why wouldn't you ? Not everything someone does on a computer is a task they wouldn't like to automate away but can't with current technology. >For everything browser based, the burden of making an API is probably relatively small It's definitely not less effort than stricking to a GUI >and if the page is simple enough, you could maybe even get away with training the AI on the page HTML and generating a response to send. Sure in special circumstances, it may be a good idea to use something else. >And for everything that’s not browser-based, I would either want the AI embedded in the software (image editors, IDEs…) or not there are all. AI embedded in software and AI operating the computer itself are entirely different things. The former is not necessarily a substitute for the latter. Having access to SORA is not at all the same thing as AI that can expertly operate Blender. And right now at least, studios would actually much prefer the latter. Even if they were equivalent (they're not), then you wouldn't be able to operate most applications without developers explicitly supporting and maintaining it first. That's infeasible.
  - agwp 2y ago
    [dead]
  - dragonwriter 2y ago
    I think the two big applications for programmatically (AI or otherwise) operating a computer via this kind of UI automation are: (1) Automated testing of apps that use traditional UIs, and (2) Automating legacy apps that it is not practical or cost effective to update.
  - simonw 2y ago
    Google really don't want to provide a useful API to their search results.
layer8 2y ago
A general-purpose assistant should be able to perform general-purpose operations, meaning the same things people do on their computers, and without having to supply a special-purpose AI-compatible interface for every single function the AI might need to operate. The AI should be able to operate any interface a human can operate.
Workaccount2 2y ago
Anthropic is selling a product to people, not software engineers.
bubaumba 2y ago
How about GUI and API are just interfaces, the rest is the same. They can coexist in the same model or setup. Same functionality can be implemented usually by one or another. In advanced products by both. But model's thinking is probably the same, it operates with concepts, arranged in sort of graph. Just my guess, have no proof and likely it's not always true.

voiper1 2y ago

Sure, it's more effecient to have it use an API. And people have been integrating those for the last while. But there's tons of applications that are locked behind a website or deckstop GUI with no API that are accessible via vision.

famouswaffles 2y ago

The vast majority of Applications cannot be used by anything other than a GUI. We built computers to be used by humans and humans overwhelmingly operate computers with GUIs. So if you want a machine that can potentially operate computers as well as humans then you're going to have to stick to GUIs. It's the same reason we're trying to build general purpose robots in a human form factor. The fact that a car is about as wide as a two horse drawn carriage is also no coincidence. You can't ignore existing infrastructure.

echoangle 2y ago
But I don’t want an AI to „operate a computer“… maybe I’m missing the point of this but I just can’t imagine a usecase where this is a good solution. For everything browser based, the burden of making an API is probably relatively small and if the page is simple enough, you could maybe even get away with training the AI on the page HTML and generating a response to send. And for everything that’s not browser-based, I would either want the AI embedded in the software (image editors, IDEs…) or not there are all.
- famouswaffles 2y ago
  >But I don’t want an AI to „operate a computer“… You don't and that's fine but certainly many people are interested in such a thing. >maybe I’m missing the point of this but I just can’t imagine a usecase where this is a good solution. If it could operate computers robustly and reliabily then why wouldn't you ? Not everything someone does on a computer is a task they wouldn't like to automate away but can't with current technology. >For everything browser based, the burden of making an API is probably relatively small It's definitely not less effort than stricking to a GUI >and if the page is simple enough, you could maybe even get away with training the AI on the page HTML and generating a response to send. Sure in special circumstances, it may be a good idea to use something else. >And for everything that’s not browser-based, I would either want the AI embedded in the software (image editors, IDEs…) or not there are all. AI embedded in software and AI operating the computer itself are entirely different things. The former is not necessarily a substitute for the latter. Having access to SORA is not at all the same thing as AI that can expertly operate Blender. And right now at least, studios would actually much prefer the latter. Even if they were equivalent (they're not), then you wouldn't be able to operate most applications without developers explicitly supporting and maintaining it first. That's infeasible.
- agwp 2y ago
  [dead]
- dragonwriter 2y ago
  I think the two big applications for programmatically (AI or otherwise) operating a computer via this kind of UI automation are: (1) Automated testing of apps that use traditional UIs, and (2) Automating legacy apps that it is not practical or cost effective to update.
- simonw 2y ago
  Google really don't want to provide a useful API to their search results.

layer8 2y ago

A general-purpose assistant should be able to perform general-purpose operations, meaning the same things people do on their computers, and without having to supply a special-purpose AI-compatible interface for every single function the AI might need to operate. The AI should be able to operate any interface a human can operate.

Workaccount2 2y ago

Anthropic is selling a product to people, not software engineers.

bubaumba 2y ago

How about GUI and API are just interfaces, the rest is the same. They can coexist in the same model or setup. Same functionality can be implemented usually by one or another. In advanced products by both. But model's thinking is probably the same, it operates with concepts, arranged in sort of graph. Just my guess, have no proof and likely it's not always true.