22 ms·
Notes on Anthropic's Computer Use Ability
- martythemaniak 2y agoI've been been hacking on a web browsing agent the last few weeks and it's given me some decent understanding of what it'd take to get this working. My approach has been to make it general-purpose enough so that I describe the mechanics of surfing the web, without building in specific knowledge about tasks or website. Some things I've learned. 1. Pixels and screenshots (video really) and keyboard/mouse events is definitely the purest and most proper way to get agents working in the long term, but it's not practical today. Cost and speed are big obvious issues, but accuracy is also low. I found that GTP4o (08-06) is just plain bad at coordinates and bounding boxes and naively feeding it screenshots just doesn't work. As a practical example, another comment mentions trying to get a list of flight recommendations from Claude computer use and it costing $5, if my agent is up for that task (haven't tested this), it would cost $0.10-$0.25. 2. "feature engineering" helps a lot right now. Explicitly highlighting things and giving the model extra context and instructions on how to use that context, how to augment the info it sees on screenshots etc. It's hard to understand things like hover text, show/hide buttons, etc from pure pixels. 3. You have to heavily constrain and prompt the model to get it to do the right thing now, but when it does it, it feels magic. 4. It makes naive, but quite understandable mistakes. The kinds of mistakes a novice user might make and it seems really hard to get this working. A mechanism to correct itself and learn is probably the better approach rather than trying to make it work right from the get-go in every situation. Again, when you see the agent fail, try again and succeed the second time based on the failure of the previous action, it's pretty magical. The first time it achieved its objective, I just started laughing out loud. I don't know if I've ever laughed at a program I've written before. It's been very interesting working on this. If traditional software is like building legos, this one is more like training a puppy. Different, but still fun. I also wonder how temporary this type of work is, I'm clearly doing a lot of manual work to augment the model's many weaknesses, but also models will get substantially better. At the same time, I can definitely see useful, practical computer use from model improvements being 2-3 years away.
- VBprogrammer 2y agoWell, this just opened up a new phase in the captcha wars.
- sunilkumardash9 2y agoIt certainly did; it's like a Pandora's box. Unless they lobotomize, we can expect Qwen and Deepseek to release open-source models.
- jazzyjackson 2y agoCaptchas were already outsourced to cheap labor, maybe 10 or 20 cents a pop? AI using image interpretation is not any cheaper so the captchas efficacy is unchanged
- deleted 2y ago[deleted]
- cl42 2y agoI really, really like this new product/API offering. Still crashes quite a bit for me and obviously makes mistakes, but shows what's possible. For the folks who are more savvy on the Docker / Linux front... 1. Did Anthropic have to write its own "control" for the mouse and keyboard? I've tried using `xdotool` and related things in the past and they were very unreliable. 2. I don't want to dismiss the power and innovation going into this model, but... (a) Why didn't Adept or someone else focused on RPA build this? (b) How much of this is standard image recognition and fine-tuning a vision model to a screen, versus something more fundamental?
- ko_pivot 2y agoAt the end of the day, the fundamental dynamic here is human creativity. We are taking a tool, the LLM, and stretching it to its limit. That’s great, but that doesn’t mean we are close to AGI. It means we are AGI.
- sunilkumardash9 2y agoThis is a step towards a human-machine hybrid world. Putting a human in the loop can do wonders. Sure, it is expensive now, but the subsequent iterations will crush it.
- im3w1l 2y agoHave you heard of Centaur chess? A human and a machine would team up to find the best chess moves against another similar team. It's not a thing anymore. Computers have advanced so much that humans can't really contribute in any meaningful sense.
- bamboozled 2y agoIs the point of your comment to make people feel depressed ? Either we're going to use these tools to augment our abilities or basically just become wiped out, at least our jobs will be, and there is no plan to provide support for anyone. Maybe the tech will make the transition to a post employment world so swift we don't even feel any negative economic effects at all, but let's see.
- exe34 2y agois this a cry for help? there's always alcohol and drugs, they can't take that away from us! (unless the robots of the future are like Bender)
- bamboozled 2y agoI'm not saying I am depressed, but I mean, the comment just sounded like such a major downer.
- Karthikeya 2y agodidn't see anyone actually going into production with any of this stuff, man, this hype cycle just continues.
- Kiro 2y agoWhat is "this stuff"? Computer use was released 3 days ago and I would say the opposite is true for LLMs in general: it's overused in production and shoehorned into stuff that doesn't need it.
- utkarsh-dixit 2y agoThis is such a idiotic hype-cycle, they just fine-tuned a model over vision API. I really don't understand why everyone is loosing their mind over this
- Mistletoe 2y agoBecause this is the last thing we can think of right now and after this is an abyss for the stock market that everyone knows is inevitable, but thinks we can avoid.
- soham123 2y agoThey showed a really cool literal example of what's coming. it's almost a chatgpt like movement.
- diffeomorphism 2y agoWhich one? The article has four examples, none of which are particularly "cool" or impressive. If anything, the examples involving moving the mouse to the address bar or getting csv's of results are very poor examples, because we can already do that much better without "computer use".
- namanyayg 2y agoWhat are some good use cases for this? Something that a business can be built around
- bicijay 2y agoA lot of automations around softwares without APIs. QA...
- deleted 2y ago[deleted]
- infecto 2y agoHave you seen UiPath? The enterprise automation business is massive.
- conception 2y agoThere are dozens of RPA businesses. UIPath etc. We had one that was a simple download from here and login and upload there. Having the accounting team be able to automate that versus devops is huge.
- nerdix 2y agoIt's main use case is making the average office worker feel the same existential dread that some programmers feel when they see a LLM spit out a bunch of code in mere seconds.
- jampekka 2y agoAnd when having to pick up the pieces when someone actually uses the thing.
- apwell23 2y agoThats the real dread
- Slump 2y agoYou're not far off. Anecdotal but I shared the Anthropic demo video and a few articles in a company slack and a lot of PM's/admin folks that are only tangentially aware of LLM powered use cases at this point shared that sentiment. Welcome to the party folks!
- acrooks 2y agoI've built a couple of experiments using it so far and it has been really interesting. On one hand, it has really helped me with prototyping incredibly fast. On the other, it is prohibitively expensive today. Essentially you pay per click, in some cases per keystroke. I tried to get it to find a flight for me. So it opened the browser, navigated to Google Flights, entered the origin, destination etc. etc. By the time it saw a price, there had already been more than a dozen LLM calls. And then it crashed due to a rate limit issue. By the time I got a list of flight recommendations it was already $5. But I think this is intended to be an early demo of what will be possible in the future. And they were very explicit that it's a beta: all of this feedback above will help them make it better. Very quickly it will get more efficient, less expensive, more reliable. So overall I'm optimistic to see where this goes. There are SO many applications for this once it's working really well.
- danielbln 2y agoHaiku 3.5 wol be here soon, and will before long support tool use and vision, so that should help a lot with cost.
- inquisitor27552 2y agotime is also a huge factor on this one, should be a nice metric god the future is here haha
- steveBK123 2y agoI guess I'm confused there's even a use case there. It's like "let me google that for you". I mean Siri can return me search results for flights. A real killer app would be something that is adaptive and smart enough to deal with all the SEO/walled gardens in the travel search space, actually understanding the airlines available and searching directly there as well as at aggregators. It could also be integrated with your Airline miles accounts and all suggested options to use miles/miles&cash/cash, etc. All of that is far more complex than .. clicking around google flights on your behalf and crashing. Further, the real killer app is that it is bullet proof enough that you entrust it to book said best flight for you. This requires getting the product to 99.99% rather than the perpetual 70-80% we are seeing all these LLM use cases hit.
- guzik 2y agoI’m not sure if anyone else has really tried, but I’ve tested it a few times and never hit meaningful results. 1) I tried using it for QA for my SaaS but agent failed multiple times to fill out a simple form, ending with it saying the task was successfully completed. 2) It couldn’t scrape contact information from a website where the details weren’t even that hidden. 3) I also tried sending a message on Discord, but it refused, saying it couldn’t do so on someone else's behalf. I mean, I’m excited for what the future holds, but right now, it’s not even in beta.
- _heimdall 2y agoI'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those tools is to act as a middle man to programmatically interpret and interact with what's on screen.
- danielbln 2y agoI think the reason is that this is the most general implementation. It doesn't need playwright or have access to the DOM or anything else, if it has a screen and mouse/keyboard, then it will work. That's quite powerful (if slow and pricey, at the moment).
- _heimdall 2y agoUnless I'm mistaken, playwright doesn't actually use the accessibility tree directly. It does have quite a few APIs for accessing nodes based on a11y attributes, but I could have sworn those were glorified query selectors rather than directly accessing the accessibility tree. Last I checked on it, maybe a year ago, there were browser proposals for standardizing the accessibility tree APIs but they were very early discussions and seemed pretty well stuck. That would be a good reason for Anthropic using image processing here though, short of forking open source a11y tools there may not have been a simple way to use accessibility data to interact.
- infecto 2y agoThose sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.
- ryukafalz 2y agoAccessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.
- sheepscreek 2y agoAppreciate the TL;DR as I got what I was looking for. Burning $30 for just trying it out doesn’t make it sound so promising at the moment.
- nilstycho 2y agoIt seems like a cheaper intermediate capability would be to give Claude the ability to SSH to your computer or to a cloud container. That would unlock a lot of possibilities, without incurring the cost of the vision model or the difficulty of cursor manipulation. Does this already exist? If not, would the benefits be lower than I think, or would the costs be higher than I think?
- reportgunner 2y agoNot just benefits, costs but also the risks are to be considered here I think.
- nilstycho 2y agoWhat are the risks? Isn't this a strict subset of the risks of full desktop access? Claude can just open a GUI terminal with Computer Use. (I think.)
- reportgunner 2y agoSoftware posing as Claude that is actually a malware tricking an unsuspecting non-terminal user into executing it is what I was thinking about.
- ActionHank 2y agoAnecdotal, but I think if you mention it in any discussions in a corporate alarm bells will go off because HACKERS use SSH. This _seems_ more like a normal user so clearly could not do anything nefarious. /s
- deleted 2y ago[deleted]
- kordlessagain 2y agoI’m working on Webwright which presents as a shell. It’s on GitHub.
- belval 2y agoThe product I would like to see out of this is a way to automate UI QA. Ideally it would be given a persona and a list of use cases, try to accomplish each task and save the state where you/it failed. Something like a Chrome lighthouse but for usability. Bonus point if it can highlight what part of my documentation is using mismatched terminology making it difficult for newcomers to understand what button I am referring to.
- steveBK123 2y agoI've seen similar sentiment even pre-LLM that AI would help automate other forms of testing, and I just don't quite see it. Implementing tests is not the hard part. You could make that an intern project or hire a consultant for 3 months. The hard part is the interpretation of results. That is - making a thing that spits out tickets/alerts is easy. The signal/noise tuning and actual investigation workflows are the hard part and still very manual & human operated. I don't see LLM mouse/keyboard control changing that yet.
- belval 2y ago> making a thing that spits out tickets/alerts is easy. I don't really believe that what I am asking for is hard, yet I still can't buy it as far as I know. > actual investigation workflows are the hard part and still very manual & human operated. Sure but it would allow your QA worker to have pre-tested usecase-based path with some flag on whether or not they may be problematic with a screen-recording and some timestamp of where it went wrong. These will always need human-in-the-loop to vet the findings before cutting a ticket to development team.
- steveBK123 2y agoFair - I'm not personally familiar with state of the art in UI QA automation, but I know theres been various screen recording type tools available for a decade+ with mixed success. I come more from a "big data" background, and have dealt with CTOs who think "can't we just use AI?" is the answer to data quality checking multi-PB data lakes with 1000s of unique datasets from 100s of vendors. That is - they don't want to staff a data quality team, they think you can just magic it all away. The answer was always - sure, but you are fixated on the easy part - anomaly detection. Actual data analysis on what broke, when, how, why, and escalating to data provider was always 95% of the work. Someone needs to look at the exhaust, and there will be exhaust every single day.. so you can kill your dev teams productivity or actually staff an operations team responsible for the tickets the thing spits out.
- alt-glitch 2y agoI wonder if I can hook up `scrcpy` with this and give it control over an Android. Can it drag the mouse? That'd be needed to navigate the phone at least.
- elif 2y agoFor my home automation system my aspiration is to give the AI control of an android virtual machine
- Jayakumark 2y agoAny idea on how does Sonnet does this, is the image annotated with bounding boxes on text boxes etc. along with its coordinates before sending to sonnet and it responds with box name back or co-ordinate back or ? is SAM2 used for segmenting everything before sending to sonnet ?
- sys32768 2y agoSo robotic process automation gains intelligence and we can train an AI intern to assist with tasks. Our own personal digital squire. Then eventually we become assistants to AI.
- azinman2 2y agoI saw those demoed yesterday. The model was asked to create a cool visualization. It ultimately tried to install steamlit and go its page, only to find its own Claude software running streamlit, so as part of debugging it killed itself. Not ready to let that go wild on my own computer!
- bonoboTP 2y agoThis kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transaction, it will only buy what's needed to accomplish the task. There will be enormous push towards steering these software agents towards similarly shady practices instead of making them act in the true interest of the user. The ads will be built into the weights of the model or something.
- tracerbulletx 2y agoAds will move to the layer of the new interface when that happens. Also a computer can't watch a youtube video for you or look at funny cat pictures. You can still put ads next to things people want to look at.
- lupusreal 2y agoOnly a matter of time before computers are generating videos of cats cuter than any real cat.
- rty32 2y agoCare to elaborate on the idea? I suppose you mean that ads will come to this "computer use" tool itself. Now, will users keep it in the foreground, when they already expect the tool to do (almost) everything for them?
- steveBK123 2y agoI think the point is - don't be so naive. Companies are investing near trillions into developing models, training models, compute, datacenter, nuclear reactors, etc. Is the endgame some free/cheap tool that abstracts away the entire ad based web economy to the benefit of end users? Imagine something closer to a super duper smart useful Siri/Alexa that feeds you product recommendations, paid placement, and other ads interspersed with your actual request response. Hey Siri what temperature is it? It's 45 and going to be chilly today, a North Face jacket might be handy today.. can I recommend you a few models? What's your size?
- elif 2y agoIf it's only downside is cost, and cost is prohibitively expensive for all practical uses, Why didn't this project start with https://huggingface.co/meta-llama/Llama-3.2-11B-Vision https://huggingface.co/meta-llama/Llama-3.2-11B-Vision
- imranq 2y agoThis is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.
- supriyo-biswas 2y agoI could see this coming into Apple Intelligence for example; you could simply ask the browser to buy stuff off of your favorite store, or even do a chain of tasks like informing a contact off your list that you've bought said thing, etc. The possibilities are quite exciting, in fact, even though the technology isn't quite there yet.
- imranq 2y agoApple should hook into app functions themselves instead of relying on UI. I would be really surprised if Apple made a browser automation tool, since that would be the complete opposite of the "it just works" credo
- deleted 2y ago[deleted]
- danielbln 2y agoThere is plenty of legacy software out there that has no and will never have a nice API to integrate with. Those are the situations where the terrible solutions are either let a human do it or automate the human tool chain from a high level. This is the LLM spin on it. Is it an efficient or even good solution? Hell no, but if there is no other solution to automation (assuming that's the goal) then does that matter?
- imranq 2y agoI'd argue that this is not even a solution to begin with. If the LLM gets even one pixel value wrong, then at best the whole process breaks down. At worst, you could do some irreversible damage.