12 ms·

Gemini 2.5 Computer Use model

636 points by mfiguiere 1y ago

susu12138 1y ago
[flagged]
- susu12138 1y ago
  It will increase gemini cli ability.
ihsw 1y ago
[dead]
xnx 1y ago
I've had good success with the Chrome devtools MCP (https://github.com/ChromeDevTools/chrome-devtools-mcp https://github.com/ChromeDevTools/chrome-devtools-mcp) for browser automation with Gemini CLI, so I'm guessing this model will work even better.
- arkmm 1y ago
  What sorts of automations were you able to get working with the Chrome dev tools MCP?
  - odie5533 1y ago
    Not OP, but in my experience, Jest and Playwright are so much faster that it's not worth doing much with the MCP. It's a neat toy, but it's just too slow for an LLM to try to control a browser using MCP calls.
    - atonse 1y ago
      Yeah I think it would be better to just have the model write out playwright scripts than the way it's doing it right now (or at least first navigate manually and then based on that, write a playwright typescript script for future tests). Cuz right now it's way too slow... perform an action, then read the results, then wait for the next tool call, etc.
      - omneity 1y ago
        This is basically our approach with Herd[0]. We operate agents that develop, test and heal trails[1, 2], which are packaged browser automations that do not require browser use LLMs to run and therefore are much cheaper and reliable. Trail automations are then abstracted as a REST API and MCP[3] which can be used either as simple functions called from your code, or by your own agent, or any combination of such. You can build your own trails, publish them on our registry, compose them ... You can also run them in a distributed fashion over several Herd clients where we take care of the signaling and communication but you simply call functions. The CLI and npm & python packages [4, 5] might be interesting as well. Note: The automation stack is entirely home-grown to enable distributed orchestration, and doesn't rely on puppeteer nor playwright but the browser automation API[6] is relatively similar to ease adoption. We also don't use the Chrome Devtools Protocol and therefore have a different tradeoff footprint. 0: https://herd.garden https://herd.garden 1: https://herd.garden/trails https://herd.garden/trails 2: https://herd.garden/docs/trails-automations https://herd.garden/docs/trails-automations 3: https://herd.garden/docs/reference-mcp-server https://herd.garden/docs/reference-mcp-server 4: https://www.npmjs.com/package/@monitoro/herd https://www.npmjs.com/package/@monitoro/herd 5: https://pypi.org/project/monitoro-herd/ https://pypi.org/project/monitoro-herd/ 6: https://herd.garden/docs/reference-page https://herd.garden/docs/reference-page
        atonse 1y ago
        Whoa that’s cool. I’ll check it out, thanks!
        omneity 1y ago
        Thanks! Let me know if you give it a shot and I’ll be happy to help you with anything.
        jarek83 1y ago
        You might want to change column title colors as they're not visible (I can see them when highlighting the text) https://herd.garden/docs/alternative-herd-vs-puppeteer/ https://herd.garden/docs/alternative-herd-vs-puppeteer/
    - typpilol 1y ago
      You can use it for debugging with the llm though.
      - rs186 1y ago
        In theory or in practice?
        typpilol 1y ago
        I've used it a few times with vscode. Although I find the electron mcp is better for what I'm doing at the moment.
    - raffraffraff 1y ago
      Actually the super power of having the LLM in the bowser may be that it vastly simplifies using LLMs to write Playwright scripts. Case in point, last week I wrote a scraper for Rate Your Music, but found it frustrating. I'm not experienced with Playwright, so I used vscode with Claude to iterate in the project. Constantly diving into devtools, copying outter html, inspecting specific elements etc is a chore that this could get around, making for faster development of complex tests
    - nsonha 1y ago
      Not tested much but Playright can read browser_network_requests' response, which is a much faster way to extract information than waiting for all the requests to finish, then parse the html, when what you're looking for is already nicely returned in an api call. Puppeteer MCP server doesn't have an equivalence.
  - grantcarthew 1y ago
    I've used it to read authenticated pages with Chromium. It can be run as a headless browser and convert the HTML to markdown, but I generally open Chromium, authenticate to the system, then allow the CLI agent to interact with the page. https://github.com/grantcarthew/scripts/blob/main/get-webpage https://github.com/grantcarthew/scripts/blob/main/get-webpag...
- iLoveOncall 1y ago
  This has absolutely nothing in common with a model for computer use... This uses pre-defined tools provided in the MCP server by Google, nothing to do with a general model supposed to work for any software.
  - falcor84 1y ago
    The general model is what runs in an agentic loop, deciding which of the MCP commands to use at each point to control the browser. From my experimentation, you can mix and match between the model and the tools available, even when the model was tuned to use a specific set of tools.
- informal007 1y ago
  Computer use model comes from interactive demand with computer automatically, Chrome devtools MCP might be one of the core pushers.
strangescript 1y ago
I assume its tool calling and structured output are way better, but this model isn't in Studio unless its being silently subbed in.
- phamilton 1y ago
  Just tried it in an existing coding agent and it rejected the requests because computer tools weren't defined.
  - omkar_savant 1y ago
    We can definitely make the docs more clear here but the model requires using the computer_use tool. If you have custom tools, you'll need to exclude predefined tools if they clash with our action space. See this section: https://googledevai.devsite.corp.google.com/gemini-api/docs/computer-use#custom-functions https://googledevai.devsite.corp.google.com/gemini-api/docs/... And the repo has a sample setup for using the default computer use tool: https://github.com/google/computer-use-preview https://github.com/google/computer-use-preview
cryptoz 1y ago
Computer Use models are going to ruin simple honeypot form fields meant to detect bots :(
- layman51 1y ago
  You mean the ones where people add a question that is like "What is 10+3?"
- jebronie 1y ago
  I just tried to submit a contact form with it. It successfully solved the ReCaptcha but failed to fill in a required field and got stuck. We're safe.
phamilton 1y ago
It successfully got through the captcha at https://www.google.com/recaptcha/api2/demo https://www.google.com/recaptcha/api2/demo
- siva7 1y ago
  probably because its ip is coming from googles own subnet
  - asadm 1y ago
    isnt it coming from browserbase container?
    - ripbozo 1y ago
      Interestingly the IP I got when prompting `what is my IP` was `73.120.125.54` - which is a residential comcast IP.
      - martinald 1y ago
        Looks like browserbase has proxies, which will be often residential IPs.
- jampa 1y ago
  The automation is powered through Browserbase, which has a captcha solver. (Whether it is automated or human, I don't know.)
  - peytoncasper 1y ago
    We do not use click farms! You should check out our most recent announcement about Web Bot Auth https://www.browserbase.com/blog/cloudflare-browserbase-pioneering-identity https://www.browserbase.com/blog/cloudflare-browserbase-pion...
- simonw 1y ago
  Post edited: I was wrong about this. Gemini tried to solve the Google CAPTCHA but it was actually Browserbase that did the solve, notes here: https://simonwillison.net/2025/Oct/7/gemini-25-computer-use-captchas/ https://simonwillison.net/2025/Oct/7/gemini-25-computer-use-...
dude250711 1y ago
Have average Google developers been told/hinted that their bonuses/promotions will be tied to their proactivity in using Gemini for project work?
- peddling-brink 1y ago
  > bonuses/promotions more like continued employment.
  - astrange 1y ago
    FAANG much prefers to not pay you and let you leave on your own.
- teaearlgraycold 1y ago
  I know there was a memo telling Googlers they are expected to use AI at work and it’s expected for their performance to increase as a result.
  - dude250711 1y ago
    The HBO's Silicon Valley ended way too soon. The plot pretty much writes itself.
    - Imustaskforhelp 1y ago
      Don't worry Maybe someone will create AI slop for this on Sora 2 or the likes (this was satire) On a serious note, What the fuck is happening in the world.
password54321 1y ago
doesn't seem like it makes sense to train AI around human user interfaces which aren't really efficient. It is like building a mechanical horse.
- pixl97 1y ago
  Right, let's make APIs for everything... [Looks around and sees people not making APIs for everything] Well that didn't work.
  - odie5533 1y ago
    Every website and application is just layers of data. Playwright and similar tools have options for taking Snapshots that contain data like text, forms, buttons, etc that can be interacted with on a site. All the calls a website makes are just APIs. Even a native application is made up of WinForms that can be inspected.
    - pixl97 1y ago
      Ah, so now you're turning LLMs into web browsers capable of parsing Javascript to figure out what a human might be looking at, let's see how many levels deep we can go.
      - measurablefunc 1y ago
        Just inspect the memory content of the process. It's all just numbers at the end of the day & algorithms do not have any understanding of what the numbers mean other than generating other numbers in response to the input numbers. For the record I agree w/ OP, screenshots are not a good interface for the same reasons that trains, subways, & dedicates lanes for mass transit are obviously superior to cars & their associated attendant headaches.
        ssl-3 1y ago
        Maybe some day, sure. We may eventually live in a utopia where everyone has quick, efficient, accessible mass transit available that allows them to move between any two points on the globe with unfettered grace. That'd be neat. But for now: The web exists, and is universal. We have programs that can render websites to an image in memory (solved for ~30 years), and other programs that can parse images of fully-rendered websites (solved for at least a few years), along with bots that can click on links (solved much more recently). Maybe tomorrow will be different.
ramoz 1y ago
This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance. Obviously much harder with UI vs agent events similar to the below. https://docs.claude.com/en/docs/claude-code/hooks https://docs.claude.com/en/docs/claude-code/hooks https://google.github.io/adk-docs/callbacks/ https://google.github.io/adk-docs/callbacks/
- peytoncasper 1y ago
  Hi! I work in identity products at Browserbase. I’ve spent a fair amount of time lately thinking about how to layer RBAC across the web. Do you think callbacks are how this gets done?
  - ramoz 1y ago
    Disclaimer: Im a cofounder, we focus critical spaces with AI. Also i was the feature request for claude code hooks. But my bet - we will not deploy a single agent into any real environment without deterministic guarantees. Hooks are a means... Browserbase with hooks would be really powerful, governance beyond RBAC (but of course enabling relevant guardrailing as well - "does agent have permission to access this sharepoint right now, within this context, to conduct action x?"). I would love to meet with you actually, my shop cares intimately about agent verification and governance. Soon to release the tool I originally designed for claude code hooks.
    - peytoncasper 1y ago
      Let’s chat my email is peyton at browserbase dot com
- serf 1y ago
  >This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance. knowing how many times Claude Code breezed through a hook call and threw it away after actually computing the hook for an answer and then proceeding to not integrate the hook results ; I think the concept of 'governance' is laughable. LLMs are so much further from determinism/governance than people seem to realize. I've even seen earlier CC breeze through a hook that ends with a halting test failure and "DO NOT PROCEED" verbage. The only hook that is guaranteed to work on call is a big theoretical dangerous claude-killing hook.
CuriouslyC 1y ago
I feel like screenshots should be the last thing you reach for. There's a whole universe of data from accessibility subsystems.
- ekelsen 1y ago
  and all sorts of situations where they don't work. When they do work it's great, but if they don't and you rely on them, you have nothing.
  - CuriouslyC 1y ago
    Oh yeah, using all available data channels in proportion to their cost and utility is the right choice, 100%.
- bonoboTP 1y ago
  The rendered visual layout is designed in a way to be spatially organized perceptually to make sense. It's a bit like PDFs. I imagine that the underlying hierarchy tree can be quite messy and spaghetti, so your best bet is to use it in the form that the devs intended and tested it for. I think screenshots are a really good and robust idea. It bothers the more structured-minded people, but apps are often not built so well. They are built until the point that it looks fine and people are able to use it. I'm pretty sure people who rely on accessibility systems have lots of complaints about this.
  - CuriouslyC 1y ago
    The progressives were pretty good at pushing accessibility in applications, it's not perfect but every company I've worked with since the mid 2010s has made a big todo about accessibility. For stuff on linux you can instrument observability in a lot of different ways that are more efficient than screenshots, so I don't think it's generally the right way to move forward, but screenshots are universal and we already have capable vision models so it's sort of a local optimization move.
- nicman23 1y ago
whinvik 1y ago
My general experience has been that Gemini is pretty bad at tool calling. The recent Gemini 2.5 Flash release actually fixed some of those issues but this one is Gemini 2.5 Pro with no indication about tool calling improvements.
TIPSIO 1y ago
Painfully slow
- John7878781 1y ago
  That doesn't matter so much when it can happen in the background.
  - alganet 1y ago
    It matters a lot for E2E testing. I would totally replace the ease of the AI solution for a faster, more complicated one if it starts impacting build times. Few things are more frustrating for a team than maintaining a slow E2E browser test suite.
Oras 1y ago
It is actually quite good at following instructions, but I tried clicking on job application links, and since they open in a new window, it couldn't find the new window. I suppose it might be an issue with BrowserBase, or just the way this demo was set up.
- MiguelG719 1y ago
  are you running into this issue on gemini.browserbase.com or the google/computer-use-preview github repo?
  - Oras 1y ago
    on gemini.browserbase.com
mianos 1y ago
I sure hope this is better than pathetically useless. I assume it is to replace the extremely frustrating Gemini for Android. If I have a bluetooth headset and I try "play music on Spotify" it fails about half the time. Even with youtube music. I could not believe it was so bad so I just sat at my desk with the helmet on and tried it over and over. It seems to recognise the speech but simply fails to do anything. Brand new Pixel 10. The old speech recognition system was way dumber but it actually worked.
- bsimpson 1y ago
  I was riding my motorcycle the other day, and asked my helmet to "call <friend>." Gemini infuriatingly replied "I cannot directly make calls for you. Is there something else I can help you with?" This absolutely used to work. Reminds me of an anecdote where Amazon invested howevermany personlives in building AI for Alexa, only to discover that alarms, music, and weather make up the large majority of things people actually use smart speakers for. They're making these things worse at their main jobs so they can sell the sizzle of AI to investors.
  - mianos 1y ago
    Yes, I am also talking about a Cardo. If it didn't used to work near 100% of the time this time last year it might not be so incredibly annoying, but to go from working to complete crap with no choice to be able to go back to the working system is bad. It's like google staff are saying "If it means promotion, we don't give a shit about users".
  - krotton 1y ago
    I remember trying "call <my wife's name as in my contacts>" a few years ago and Google Assistant cheerfully responding with "calling <first Google search hit with the same name>, doctor". I couldn't believe it, but back then, instead of searching my contact list, it searched the web and called the first phone number it found. A few years later (but still pre-Gemini), I tried again and it worked as expected. Now, some time ago, post-Gemini, it refused to make a call. This is basically the first most obvious kind of voice command that comes to mind when wondering what you can do with the assistant on your phone and it's still (again?) not working after years of voice assistant development. Astonishing.
mosura 1y ago
One of the slightly buried stories here is BrowserBase themselves. Great stuff.
bonoboTP 1y ago
There are some absolutely atrocious UIs out there for many office workers, who spend hours clicking buttons opening popup after popup clicking repetitively on checkboxes etc. E.g. entering travel costs or somesuch in academia and elsewhere. You have no idea how annoying that type of work is, you pull out your hair. Why don't they make better UIs, you ask? If you ask, you have no idea how bad things are. Because they don't care, there is no communication, it seems fine, the software creators are hard to reach, the software is approved by people who never used it and decide based on gut feel, powerpoints and feature tickmarks. Even big name brands are horrible at this, like SAP. If such AI tools allow to automate this soulcrushing drudgery, it will be great. I know that you can technically script things Selenium, AutoHotkey whatnot. But you can imagine that it's a nonstarter in a regular office. This kind of tool could make things like that much more efficient. And it's not like it will then obviate the jobs entirely (at least not right away). These offices often have immense backlogs and are understaffed as is.
numpad0 1y ago
How big are Gemini 2.5(Pro/Flash/Lite) models in parameter counts, in experts' guesstimation? Is it towards 50B, 500B, or bigger still? Even Flash feels smart enough for vibe coding tasks.
- thomasm6m6 1y ago
  2.5 Flash Lite replaced 2.0 Flash Lite which replaced 1.5 Flash 8B, so one might suspect 2.5 Flash Lite is well under 50B
jcims 1y ago
(Just using the browserbase demo) Knowing it's technically possible is one thing, but giving it a short command and seeing it go log in to a site, scroll around, reply to posts, etc. is eerie. Also it tied me at wordle today, making the same mistake I did on the second to lass guess. Too bad you can't talk to it while it's working.
iAMkenough 1y ago
Not great at Google Sheets. Repeatedly overwrites all previous columns while trying to populate new columns. > I am back in the Google Sheet. I previously typed "Zip Code" in F1, but it looks like I selected cell A1 and typed "A". I need to correct that first. I'll re-type "Zip Code" in F1 and clear A1. It seems I clicked A1 (y=219, x=72) then F1 (y=219, x=469) and typed "Zip Code", but then maybe clicked A1 again.
- omkar_savant 1y ago
  Could you share your prompt? We'll look into this one
asadm 1y ago
This is great. Now I want it to run faster than I can do it.
- pbhjpbhj 1y ago
  Then it will be detected and blocked...
omkar_savant 1y ago
Hey - I'm on the team that launched this. Please let me know if you have any questions!
- SoKamil 1y ago
  How are you going to deal with reCAPTCHA and ad impressions? Sounds like a conflict of interest.
  - omkar_savant 1y ago
    No easy answers on this one unfortunately, lots of conversations ongoing on these - but our default stance has been to hand back control to the user in cases of captcha and have them solve these when they arise.
    - qingcharles 1y ago
      What about when all your competitors are solving the CAPTCHAs?
- Awesomedonut 1y ago
  Really cool stuff! Any interesting challenges the team ran into while developing it?
- sumedh 1y ago
  I am on https://gemini.browserbase.com/ https://gemini.browserbase.com/ and just click the use case mentioned on the site "Go to Hacker News and find the most controversial post from today, then read the top 3 comments and summarize the debate." It did not work, multiple times, just gets stuck after going to Hacker news.
- bonoboTP 1y ago
  It's a bit funny that I give Google Gemini a task and then it goes on the Google Search site and it gets stuck in the captcha tarpit that's supposed to block unwanted bots. But I guess Google Gemini shouldn't be unwanted for Google. Can't you ask the search team to whitelist the Gemini bot?
animanoir 1y ago
[dead]
martinald 1y ago
Interesting, seems to use 'pure' vision and x/y coords for clicking stuff. Most other browser automation with LLMs I've seen uses the dom/accessibility tree which absolutely churns through context, but is much more 'accurate' at clicking stuff because it can use the exact text/elements in a selector. Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.
- deleted 1y ago
  [deleted]
- pbhjpbhj 1y ago
  18 attempts - emulating the human HN experience when using mobile. Well, assuming it hit other links it didn't intend to anyway. /jk
John7878781 1y ago
[dead]
dekhn 1y ago
Many years ago I was sitting at a red light on a secondary road, where the primary cross road was idle. It seemed like you could solve this using a computer vision camera system that watched the primary road and when it was idle, would expedite the secondary road's green light. This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that can detect cars passing over - trivial hardware and software - and I concluded that my approach was just far too complicated and expensive. Similarly, when I look at computers, I typically want the ML/AI system to operate on a structured data that is codified for computer use. But I guess the world is complicated enough and computers got fast enough that having an AI look at a computer screen and move/click a mouse makes sense.
- ge96 1y ago
  It's funny I'll sometimes scoot forward/rock my car but I'm not sure if it's just coincidence. Also a lot of stop lights now have that tall white camera on top.
  - bozhark 1y ago
    Like flashing lights for the first responders sensor
  - Spooky23 1y ago
    Sometimes the rocking helps with a ground loop that isn’t working well.
  - netghost 1y ago
    There's several mechanisms. The most common is (or at least was) a loop detector under the road that triggers when a vehicle is over it. Sometimes if you're not quite over it, or it's somewhat faulty that will trigger it.
- trenchpilgrim 1y ago
  FWIW those type of traffic cameras are in common use. https://www.milesight.com/company/blog/types-of-traffic-cameras https://www.milesight.com/company/blog/types-of-traffic-came...
AaronAPU 1y ago
I’m looking forward to a desktop OS optimized version so it can do the QA that I have no time for!
alexnewman 1y ago
A year ago I did something that used rag and accessibility mode to navigate ui.
realty_geek 1y ago
Absolutely hilarious how it gets stuck trying to solve captcha each time. I had to explicitly tell it not to go to google first. In the end I did manage to get it to play the housepriceguess game: https://www.youtube.com/watch?v=nqYLhGyBOnM https://www.youtube.com/watch?v=nqYLhGyBOnM I think I'll make that my equivalent of Simon Willison's "pelican riding a bicycle" test. It is fairly simple to explain but seems to trip up different LLMs in different ways.
dekhn 1y ago
I just have to say that I consider this an absolutely hilarious outcome. For many years, I focused on tech solutions that eliminated the need for a human to be in front of a computer doing tedious manual operations. For a wide range of activities, I proposed we focus on "turning everything in the world into database objects" so that computers could operate on them with minimal human effort. I spent significant effort in machine learning to achieve this. It didn't really occur to me that you could just train a computer to work directly on the semi-structured human world data (display screen buffer) through a human interface (mouse + keyboard). However, I fully support it (like all the other crazy ideas on the web that beat out the "theoretically better" approaches). I do not think it is unrealistic to expect that within a decade, we could have computer systems that can open chrome, start a video chat with somebody, go back and forth for a while to achieve a task, then hang up... with the person on the other end ever knowing they were dealing with a computer instead of a human.
- TeMPOraL 1y ago
  AI is succeeding where "theoretically better" approaches failed, because it addresses the underlying social problem. The computing ecosystem is an adversarial place, not a cooperative one. The reason we can't automate most of the tedium is by design - it's critical to how almost all money is made on the Internet. Can't monetize users when they automate your upsell channels and ad exposure away.
- ncallaway 1y ago
  > we could have computer systems that can open chrome, start a video chat with somebody, go back and forth for a while to achieve a task, then hang up... with the person on the other end ever knowing they were dealing with a computer instead of a human. Doesn't that...seem bad? I mean, it would certainly be a monumental and impressive technical accomplishment. But it still seems...quite bad to me.
  - dekhn 1y ago
    Good or bad? I don't know. It just seems inevitable.
hipassage 1y ago
hi
hipassage 1y ago
hi there
hipassage 1y ago
hi there, interesting post
deleted 1y ago
[deleted]
GeminiFan2025 1y ago
The new Gemini 2.5 model's ability to understand and interact with computer interfaces looks very impressive. It could be a game-changer for accessibility and automation. I wonder how robust it is with non-standard UI elements.
GeminiFan2025 1y ago
Impressive interface interaction by Gemini 2.5. Could be great for accessibility.
enjoylife 1y ago
> It is not yet optimized for desktop OS-level control Alas, AGI is not yet here. But I feel like if this OS-level of control was good enough, and the cost of the LLM in the loop wasn't bad, maybe that would be enough to kick start something akin to AGI.
- alganet 1y ago
  I am curious. Why do you think controlling an OS (and not just a browser) would be a move towards AGI?
  - enjoylife 1y ago
    One thought is once it can touch the underlying system, it can provision resources, spawn processes, and persist itself, crossing the line from tool to autonomous entity. I admit you could do that in a browser shell nowadays, just maybe with more restrictions and guardrails. I don’t have any strong opinions here, but I do think a lower cost to escape the walled gardens agi starts in will be a factor
    - alganet 1y ago
      I see. I guarantee you that if AGI happens, it won't happen that way. No need to worry.
  - throwaway-0001 1y ago
    Not just the os, but browsing control is enough to do 99% of the things he would want autonomously. Bank account + id + browser: has all the tools it needs to do many things: - earn money - allocate money - create accounts - delegate physical jobs to humans Create his own self loop in a server. Create a server account, use credit card + id provided, self host his own code… can now focus on getting more resources.
- pseidemann 1y ago
  Funny thing is, most humans cannot properly control a computer. Intelligence seems to be impossible to define.
mmaunder 1y ago
I prepare to be disappointed every time I click on a Google AI announcement. Which is so very unfortunate, given that they're the source of LLMs. Come on big G!! Get it together!
orliesaurus 1y ago
Does it know what's behind the "menu" of different apps? Or does it have to click on all menus and submenus to find out?
CryptoBanker 1y ago
Unless you give it specific instructions it will google something and give you the AI generated summary as the answer
mohsen1 1y ago
> Solve today's Wordle Stucks with: > ...the task is just to "solve today's Wordle", and as a web browsing robot, I cannot actually see the colors of the letters after a guess to make subsequent guesses. I can enter a word, but I cannot interpret the feedback (green, yellow, gray letters) to solve the puzzle.
- Havoc 1y ago
  So I guess it’s browsing in grey scale?
  - egeozcan 1y ago
    I tested it and gemini seems not able to solve wordle in grayscale https://g.co/gemini/share/234fb68bc9a4 https://g.co/gemini/share/234fb68bc9a4
  - daemonologist 1y ago
    It can definitely see color - I asked it to go to bing and search for the two most prominent colors in the bing background image and it did so just fine. It seems extremely lazy though; it prematurely reported as "completed" most of the tasks I gave it after the first or second step (navigating to the relevant website, usually).
    - avighnay 1y ago
      The models are mostly I believe capable of executing however as you rightly indicated 'lazy'. This 'laziness' I think is to conserve resource usage as much as possible as given the current state of AI market the infrastructure is being heavily subsidized for the user. This leads to perhaps the model being incentivized to produce an optimum result that satisfies the user by consuming the least amount of resources. This is also why most 'vibe' coding projects fail as the model is always going to give this optimum ('lazy') result by default. I have fun goading Gemini to break this ceiling when I work on my AI project - https://github.com/gingerhome/gingee https://github.com/gingerhome/gingee
- jcims 1y ago
amelius 1y ago
Only use in environments where you can roll back everything.
sbinnee 1y ago
I think it’s related that I got an email from google, titled “ Simplifying your Gemini Apps experience”. It reads no privacy maximize AI. They are going to automatically collect data from all google apps, and users no longer have options to control access to individual apps.
zomgbbq 1y ago
I would love to use this for E2E testing. It would be great to make all my assertions with high level descriptions so tests are resilient to UI changes. Seems similar to the Amazon Nova Act API which is still in research preview.
- sumedh 1y ago
  Use Playwright and some AI model to write Playwright script, running those scripts will be much faster.
- jbarber 1y ago
  This is harder than you might expect because it's hard to tell whether a passing test is a false positive (i.e. the test passed, but it should have failed). It's also hard to convey to the testing system what is an acceptable level of change in the UI - what the testing system thinks is ok, you might consider broken. There are quite a few companies out there trying to solve this problem, including my previous employer https://rainforestqa.com https://rainforestqa.com
  - nextworddev 1y ago
    LLM as judge
deleted 1y ago
[deleted]
Havoc 1y ago
At some point just having APIs for the web would just make sense. Rendering it and then throwing llms at interpreting it seems…suboptimal Impressive tech nonetheless
- guelo 1y ago
  The programmable web is a dead 20 year old dream that was crushed by the evil tech monopolists, Facebook, Google, etc. This emerging llm based automation tech is a glimmer of hope that we will be able to regain our data and autonomy.
keepamovin 1y ago
It’s basically an OODA loop. This is a good thing
btbuildem 1y ago
Does it work with ~legacy~ software? Eg, early 2000's Windows WhateverSoft's Widget Designer? Does it interface over COM? There's a goldmine to be had in automating ancient workflows that keep large corps alive.
- tech234a 1y ago
  “The Gemini 2.5 Computer Use model is primarily optimized for web browsers, but also demonstrates strong promise for mobile UI control tasks. It is not yet optimized for desktop OS-level control.”
jsrozner 1y ago
The irony is that most of tech companies make their money by forcing users to wade through garbage. For example, if you could browse the internet and avoid ads, why wouldn't you? If you could choose what twitter content to see outside of their useless algorithms, why wouldn't you?
- machiaweliczny 1y ago
  It's the same as saying if you could plunder and pillage why wouldn't you
tgsovlerkhgsel 1y ago
I'm so looking forward to it. Many of the problems that should be trivially solved with either AI or a script are hard to impossible to solve because the data is locked away in some form. Having an AI handle this may be inefficient, but as it uses the existing user interfaces, it might allow bypassing years of bureaucracy, and when the bureaucracy tries to fight it to justify its existence, it can fight it out with the EVERYONE MUST USE AI OR ELSE layers of management, while I can finally automate that idiotic task (using tens of kilowatts rather than a one-liner, but still better than having to do it by hand).
jwpapi 1y ago
Can somebody give me use cases that are faster than using an UX? How am I supposed to use this. I really can’t think of one, but I don’t want to be blind-sighted as obviously a lot of money is going into this. I also appreciate the tech behind it and functionality, but I still wonder for use cases
- omkar_savant 1y ago
  At current latency, there's a bunch of async automation usecases that one could use this for. For example: *Tedious to complete, easy to verify BPO: Filling out doctor licensing applications based on context from a web profile HR: Move candidate information from an ATS to HRIS system Logistics: Fill out a shipping order form based on a PDF of the packing label * Interact with a diverse set of sites for a single workflow Real estate: Diligence on properties that involves interacting with one of many county records websites Freight forwarding: Check the status of shipping containers across 1 of 50 port terminal sites Shipping: Post truck load requests across multiple job board sites BPO: Fetch the status of a Medicare coverage application from 1 of 50 state sites BPO: Fill out medical license forms across multiple state websites * Periodic syncs between various systems of record Clinical: Copy patient insurance info from Zocdoc into an internal system HR: Move candidate information from an ATS to HRIS system Customer onboarding: Create Salesforce tickets based on planned product installations that are logged in an internal system Logistics: Update the status of various shipments using tracking numbers on the USPS site * PDF extraction to system interaction Insurance: A broker processes a detailed project overview and creates a certificate of insurance with the specific details from the multi-page document by filling out an internal form Logistics: Fill out a shipping order form based on a PDF of the packing label Clinical: Enter patient appointment information into an EHR system based on a referral PDF Accounting: Extract invoice information from up to 50+ vendor formats and enter the details into a Google sheet without laborious OCR setup for specific formats Mortgage: Extract realtor names and address from a lease document and look up the license status on various state portals * Self healing broken RPA workflows
- viking123 1y ago
  Filling your CV in the workday applications
derekcheng08 1y ago
Really feels like computer use models may be vertical agent killers once they get good enough. Many knowledge work domains boil down to: use a web app, send an email. (e.g. recruiting, sales outreach)
- loandbehold 1y ago
  Why do you need an agent use web app through UI? Can't agent be integrated into web app natively? IMO for verticals you mentioned the missing piece is for an agent to be able to make phone calls.
  - tgsovlerkhgsel 1y ago
    Native integration, APIs etc. require the web app author to do something. A computer use agent using the UI doesn't.
albert_e 1y ago
I believe it will need very capable but small VLMs that understand common User Interfaces very well -- small enough to run locally -- paired with any other higher level models on the cloud, to achieve human-speed interactions and beyond with reliability.
xrd 1y ago
I really want this model to try userinyerface.com
clayliu 1y ago
[dead]
krawcu 1y ago
I wonder how it would behave in a scenario where it has to download some file from a shady website that has all those advertisement with fake "download"
- beepdyboop 1y ago
  haha that's a great test actually
t43562 1y ago
How ironic that words become more powerful than images.....
skc 1y ago
How likely is it that the end game becomes that we stop writing apps for actual human users and instead sites become massive walls of minified text against a black screen.
- barrenko 1y ago
  Hopefully we get entirely off the internet.
- fauigerzigerk 1y ago
  If some functionality isn't used directly by humans why not expose it as an API? If you're asking how likely it is that all human-computer interaction will take place via lengthy natural language conversations then my guess is no. Visualising information and pointing at things is just too useful to replace it with what is essentially a smart command line interface.
- password54321 1y ago
  It is all just data. It doesn't need to be rendered to become input.
- peytoncasper 1y ago
  Actually a few startups working on this! You should check out Stytch isAgent SDK. We’re partnering with them on Web Bot Auth
SilverSlash 1y ago
Is this different from ChatGPT agent mode that I can use from the web app? I found that extremely useful for my task which required running some python and javascript code with open source libraries to generate an animated video effect. I greatly appreciated ChatGPT writing the code and then running it on OpenAI's VMs instead of me pasting that code on my machine. I wish Google released something like that in AI Studio.
- nextworddev 1y ago
  with this you can use your browser on device
ChaoPrayaWave 1y ago
I've always been interested in running LLM locally to automate browser tasks, but every time I've tried, I've found the browser API to be too complex. In contrast, writing scripts directly with Playwright or Puppeteer tends to be much more stable.
saltserv 1y ago
[dead]
nsonha 1y ago
Is there a claude code for computer use models? I mean something that's actually useful and not just a claude.ai kinda thing.
Michael_Keller 1y ago
[dead]
informal007 1y ago
Future will be more challenging for fraud-detection fields, good luck for them.
Muhvmd_4 1y ago
hfhgfdfgdgg

636 points by mfiguiere 1y ago

susu12138 1y ago

[flagged]

susu12138 1y ago
It will increase gemini cli ability.

ihsw 1y ago

[dead]

xnx 1y ago

I've had good success with the Chrome devtools MCP (https://github.com/ChromeDevTools/chrome-devtools-mcp https://github.com/ChromeDevTools/chrome-devtools-mcp) for browser automation with Gemini CLI, so I'm guessing this model will work even better.

arkmm 1y ago
What sorts of automations were you able to get working with the Chrome dev tools MCP?
- odie5533 1y ago
  Not OP, but in my experience, Jest and Playwright are so much faster that it's not worth doing much with the MCP. It's a neat toy, but it's just too slow for an LLM to try to control a browser using MCP calls.
  - atonse 1y ago
    Yeah I think it would be better to just have the model write out playwright scripts than the way it's doing it right now (or at least first navigate manually and then based on that, write a playwright typescript script for future tests). Cuz right now it's way too slow... perform an action, then read the results, then wait for the next tool call, etc.
    - omneity 1y ago
      This is basically our approach with Herd[0]. We operate agents that develop, test and heal trails[1, 2], which are packaged browser automations that do not require browser use LLMs to run and therefore are much cheaper and reliable. Trail automations are then abstracted as a REST API and MCP[3] which can be used either as simple functions called from your code, or by your own agent, or any combination of such. You can build your own trails, publish them on our registry, compose them ... You can also run them in a distributed fashion over several Herd clients where we take care of the signaling and communication but you simply call functions. The CLI and npm & python packages [4, 5] might be interesting as well. Note: The automation stack is entirely home-grown to enable distributed orchestration, and doesn't rely on puppeteer nor playwright but the browser automation API[6] is relatively similar to ease adoption. We also don't use the Chrome Devtools Protocol and therefore have a different tradeoff footprint. 0: https://herd.garden https://herd.garden 1: https://herd.garden/trails https://herd.garden/trails 2: https://herd.garden/docs/trails-automations https://herd.garden/docs/trails-automations 3: https://herd.garden/docs/reference-mcp-server https://herd.garden/docs/reference-mcp-server 4: https://www.npmjs.com/package/@monitoro/herd https://www.npmjs.com/package/@monitoro/herd 5: https://pypi.org/project/monitoro-herd/ https://pypi.org/project/monitoro-herd/ 6: https://herd.garden/docs/reference-page https://herd.garden/docs/reference-page
      - atonse 1y ago
        Whoa that’s cool. I’ll check it out, thanks!
        omneity 1y ago
        Thanks! Let me know if you give it a shot and I’ll be happy to help you with anything.
        jarek83 1y ago
        You might want to change column title colors as they're not visible (I can see them when highlighting the text) https://herd.garden/docs/alternative-herd-vs-puppeteer/ https://herd.garden/docs/alternative-herd-vs-puppeteer/
  - typpilol 1y ago
    You can use it for debugging with the llm though.
    - rs186 1y ago
      In theory or in practice?
      - typpilol 1y ago
        I've used it a few times with vscode. Although I find the electron mcp is better for what I'm doing at the moment.
  - raffraffraff 1y ago
    Actually the super power of having the LLM in the bowser may be that it vastly simplifies using LLMs to write Playwright scripts. Case in point, last week I wrote a scraper for Rate Your Music, but found it frustrating. I'm not experienced with Playwright, so I used vscode with Claude to iterate in the project. Constantly diving into devtools, copying outter html, inspecting specific elements etc is a chore that this could get around, making for faster development of complex tests
  - nsonha 1y ago
    Not tested much but Playright can read browser_network_requests' response, which is a much faster way to extract information than waiting for all the requests to finish, then parse the html, when what you're looking for is already nicely returned in an api call. Puppeteer MCP server doesn't have an equivalence.
- grantcarthew 1y ago
  I've used it to read authenticated pages with Chromium. It can be run as a headless browser and convert the HTML to markdown, but I generally open Chromium, authenticate to the system, then allow the CLI agent to interact with the page. https://github.com/grantcarthew/scripts/blob/main/get-webpage https://github.com/grantcarthew/scripts/blob/main/get-webpag...
iLoveOncall 1y ago
This has absolutely nothing in common with a model for computer use... This uses pre-defined tools provided in the MCP server by Google, nothing to do with a general model supposed to work for any software.
- falcor84 1y ago
  The general model is what runs in an agentic loop, deciding which of the MCP commands to use at each point to control the browser. From my experimentation, you can mix and match between the model and the tools available, even when the model was tuned to use a specific set of tools.
informal007 1y ago
Computer use model comes from interactive demand with computer automatically, Chrome devtools MCP might be one of the core pushers.

strangescript 1y ago

I assume its tool calling and structured output are way better, but this model isn't in Studio unless its being silently subbed in.

phamilton 1y ago
Just tried it in an existing coding agent and it rejected the requests because computer tools weren't defined.
- omkar_savant 1y ago
  We can definitely make the docs more clear here but the model requires using the computer_use tool. If you have custom tools, you'll need to exclude predefined tools if they clash with our action space. See this section: https://googledevai.devsite.corp.google.com/gemini-api/docs/computer-use#custom-functions https://googledevai.devsite.corp.google.com/gemini-api/docs/... And the repo has a sample setup for using the default computer use tool: https://github.com/google/computer-use-preview https://github.com/google/computer-use-preview

cryptoz 1y ago

Computer Use models are going to ruin simple honeypot form fields meant to detect bots :(

layman51 1y ago
You mean the ones where people add a question that is like "What is 10+3?"
jebronie 1y ago
I just tried to submit a contact form with it. It successfully solved the ReCaptcha but failed to fill in a required field and got stuck. We're safe.

phamilton 1y ago

It successfully got through the captcha at https://www.google.com/recaptcha/api2/demo https://www.google.com/recaptcha/api2/demo

siva7 1y ago
probably because its ip is coming from googles own subnet
- asadm 1y ago
  isnt it coming from browserbase container?
  - ripbozo 1y ago
    Interestingly the IP I got when prompting `what is my IP` was `73.120.125.54` - which is a residential comcast IP.
    - martinald 1y ago
      Looks like browserbase has proxies, which will be often residential IPs.
jampa 1y ago
The automation is powered through Browserbase, which has a captcha solver. (Whether it is automated or human, I don't know.)
- peytoncasper 1y ago
  We do not use click farms! You should check out our most recent announcement about Web Bot Auth https://www.browserbase.com/blog/cloudflare-browserbase-pioneering-identity https://www.browserbase.com/blog/cloudflare-browserbase-pion...
simonw 1y ago
Post edited: I was wrong about this. Gemini tried to solve the Google CAPTCHA but it was actually Browserbase that did the solve, notes here: https://simonwillison.net/2025/Oct/7/gemini-25-computer-use-captchas/ https://simonwillison.net/2025/Oct/7/gemini-25-computer-use-...

dude250711 1y ago

Have average Google developers been told/hinted that their bonuses/promotions will be tied to their proactivity in using Gemini for project work?

peddling-brink 1y ago
> bonuses/promotions more like continued employment.
- astrange 1y ago
  FAANG much prefers to not pay you and let you leave on your own.
teaearlgraycold 1y ago
I know there was a memo telling Googlers they are expected to use AI at work and it’s expected for their performance to increase as a result.
- dude250711 1y ago
  The HBO's Silicon Valley ended way too soon. The plot pretty much writes itself.
  - Imustaskforhelp 1y ago
    Don't worry Maybe someone will create AI slop for this on Sora 2 or the likes (this was satire) On a serious note, What the fuck is happening in the world.

password54321 1y ago

doesn't seem like it makes sense to train AI around human user interfaces which aren't really efficient. It is like building a mechanical horse.

pixl97 1y ago
Right, let's make APIs for everything... [Looks around and sees people not making APIs for everything] Well that didn't work.
- odie5533 1y ago
  Every website and application is just layers of data. Playwright and similar tools have options for taking Snapshots that contain data like text, forms, buttons, etc that can be interacted with on a site. All the calls a website makes are just APIs. Even a native application is made up of WinForms that can be inspected.
  - pixl97 1y ago
    Ah, so now you're turning LLMs into web browsers capable of parsing Javascript to figure out what a human might be looking at, let's see how many levels deep we can go.
    - measurablefunc 1y ago
      Just inspect the memory content of the process. It's all just numbers at the end of the day & algorithms do not have any understanding of what the numbers mean other than generating other numbers in response to the input numbers. For the record I agree w/ OP, screenshots are not a good interface for the same reasons that trains, subways, & dedicates lanes for mass transit are obviously superior to cars & their associated attendant headaches.
      - ssl-3 1y ago
        Maybe some day, sure. We may eventually live in a utopia where everyone has quick, efficient, accessible mass transit available that allows them to move between any two points on the globe with unfettered grace. That'd be neat. But for now: The web exists, and is universal. We have programs that can render websites to an image in memory (solved for ~30 years), and other programs that can parse images of fully-rendered websites (solved for at least a few years), along with bots that can click on links (solved much more recently). Maybe tomorrow will be different.

ramoz 1y ago

This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance. Obviously much harder with UI vs agent events similar to the below. https://docs.claude.com/en/docs/claude-code/hooks https://docs.claude.com/en/docs/claude-code/hooks https://google.github.io/adk-docs/callbacks/ https://google.github.io/adk-docs/callbacks/

peytoncasper 1y ago
Hi! I work in identity products at Browserbase. I’ve spent a fair amount of time lately thinking about how to layer RBAC across the web. Do you think callbacks are how this gets done?
- ramoz 1y ago
  Disclaimer: Im a cofounder, we focus critical spaces with AI. Also i was the feature request for claude code hooks. But my bet - we will not deploy a single agent into any real environment without deterministic guarantees. Hooks are a means... Browserbase with hooks would be really powerful, governance beyond RBAC (but of course enabling relevant guardrailing as well - "does agent have permission to access this sharepoint right now, within this context, to conduct action x?"). I would love to meet with you actually, my shop cares intimately about agent verification and governance. Soon to release the tool I originally designed for claude code hooks.
  - peytoncasper 1y ago
    Let’s chat my email is peyton at browserbase dot com
serf 1y ago
>This will never hit a production enterprise system without some form of hooks/callbacks in place to instill governance. knowing how many times Claude Code breezed through a hook call and threw it away after actually computing the hook for an answer and then proceeding to not integrate the hook results ; I think the concept of 'governance' is laughable. LLMs are so much further from determinism/governance than people seem to realize. I've even seen earlier CC breeze through a hook that ends with a halting test failure and "DO NOT PROCEED" verbage. The only hook that is guaranteed to work on call is a big theoretical dangerous claude-killing hook.

CuriouslyC 1y ago

I feel like screenshots should be the last thing you reach for. There's a whole universe of data from accessibility subsystems.

ekelsen 1y ago
and all sorts of situations where they don't work. When they do work it's great, but if they don't and you rely on them, you have nothing.
- CuriouslyC 1y ago
  Oh yeah, using all available data channels in proportion to their cost and utility is the right choice, 100%.
bonoboTP 1y ago
The rendered visual layout is designed in a way to be spatially organized perceptually to make sense. It's a bit like PDFs. I imagine that the underlying hierarchy tree can be quite messy and spaghetti, so your best bet is to use it in the form that the devs intended and tested it for. I think screenshots are a really good and robust idea. It bothers the more structured-minded people, but apps are often not built so well. They are built until the point that it looks fine and people are able to use it. I'm pretty sure people who rely on accessibility systems have lots of complaints about this.
- CuriouslyC 1y ago
  The progressives were pretty good at pushing accessibility in applications, it's not perfect but every company I've worked with since the mid 2010s has made a big todo about accessibility. For stuff on linux you can instrument observability in a lot of different ways that are more efficient than screenshots, so I don't think it's generally the right way to move forward, but screenshots are universal and we already have capable vision models so it's sort of a local optimization move.
nicman23 1y ago

whinvik 1y ago

My general experience has been that Gemini is pretty bad at tool calling. The recent Gemini 2.5 Flash release actually fixed some of those issues but this one is Gemini 2.5 Pro with no indication about tool calling improvements.

TIPSIO 1y ago

Painfully slow

John7878781 1y ago
That doesn't matter so much when it can happen in the background.
- alganet 1y ago
  It matters a lot for E2E testing. I would totally replace the ease of the AI solution for a faster, more complicated one if it starts impacting build times. Few things are more frustrating for a team than maintaining a slow E2E browser test suite.

Oras 1y ago

It is actually quite good at following instructions, but I tried clicking on job application links, and since they open in a new window, it couldn't find the new window. I suppose it might be an issue with BrowserBase, or just the way this demo was set up.

MiguelG719 1y ago
are you running into this issue on gemini.browserbase.com or the google/computer-use-preview github repo?
- Oras 1y ago
  on gemini.browserbase.com

mianos 1y ago

I sure hope this is better than pathetically useless. I assume it is to replace the extremely frustrating Gemini for Android. If I have a bluetooth headset and I try "play music on Spotify" it fails about half the time. Even with youtube music. I could not believe it was so bad so I just sat at my desk with the helmet on and tried it over and over. It seems to recognise the speech but simply fails to do anything. Brand new Pixel 10. The old speech recognition system was way dumber but it actually worked.

bsimpson 1y ago
I was riding my motorcycle the other day, and asked my helmet to "call <friend>." Gemini infuriatingly replied "I cannot directly make calls for you. Is there something else I can help you with?" This absolutely used to work. Reminds me of an anecdote where Amazon invested howevermany personlives in building AI for Alexa, only to discover that alarms, music, and weather make up the large majority of things people actually use smart speakers for. They're making these things worse at their main jobs so they can sell the sizzle of AI to investors.
- mianos 1y ago
  Yes, I am also talking about a Cardo. If it didn't used to work near 100% of the time this time last year it might not be so incredibly annoying, but to go from working to complete crap with no choice to be able to go back to the working system is bad. It's like google staff are saying "If it means promotion, we don't give a shit about users".
- krotton 1y ago
  I remember trying "call <my wife's name as in my contacts>" a few years ago and Google Assistant cheerfully responding with "calling <first Google search hit with the same name>, doctor". I couldn't believe it, but back then, instead of searching my contact list, it searched the web and called the first phone number it found. A few years later (but still pre-Gemini), I tried again and it worked as expected. Now, some time ago, post-Gemini, it refused to make a call. This is basically the first most obvious kind of voice command that comes to mind when wondering what you can do with the assistant on your phone and it's still (again?) not working after years of voice assistant development. Astonishing.

mosura 1y ago

One of the slightly buried stories here is BrowserBase themselves. Great stuff.

bonoboTP 1y ago

There are some absolutely atrocious UIs out there for many office workers, who spend hours clicking buttons opening popup after popup clicking repetitively on checkboxes etc. E.g. entering travel costs or somesuch in academia and elsewhere. You have no idea how annoying that type of work is, you pull out your hair. Why don't they make better UIs, you ask? If you ask, you have no idea how bad things are. Because they don't care, there is no communication, it seems fine, the software creators are hard to reach, the software is approved by people who never used it and decide based on gut feel, powerpoints and feature tickmarks. Even big name brands are horrible at this, like SAP. If such AI tools allow to automate this soulcrushing drudgery, it will be great. I know that you can technically script things Selenium, AutoHotkey whatnot. But you can imagine that it's a nonstarter in a regular office. This kind of tool could make things like that much more efficient. And it's not like it will then obviate the jobs entirely (at least not right away). These offices often have immense backlogs and are understaffed as is.

numpad0 1y ago

How big are Gemini 2.5(Pro/Flash/Lite) models in parameter counts, in experts' guesstimation? Is it towards 50B, 500B, or bigger still? Even Flash feels smart enough for vibe coding tasks.

thomasm6m6 1y ago
2.5 Flash Lite replaced 2.0 Flash Lite which replaced 1.5 Flash 8B, so one might suspect 2.5 Flash Lite is well under 50B

jcims 1y ago

(Just using the browserbase demo) Knowing it's technically possible is one thing, but giving it a short command and seeing it go log in to a site, scroll around, reply to posts, etc. is eerie. Also it tied me at wordle today, making the same mistake I did on the second to lass guess. Too bad you can't talk to it while it's working.

iAMkenough 1y ago

Not great at Google Sheets. Repeatedly overwrites all previous columns while trying to populate new columns. > I am back in the Google Sheet. I previously typed "Zip Code" in F1, but it looks like I selected cell A1 and typed "A". I need to correct that first. I'll re-type "Zip Code" in F1 and clear A1. It seems I clicked A1 (y=219, x=72) then F1 (y=219, x=469) and typed "Zip Code", but then maybe clicked A1 again.

omkar_savant 1y ago
Could you share your prompt? We'll look into this one

asadm 1y ago

This is great. Now I want it to run faster than I can do it.

pbhjpbhj 1y ago
Then it will be detected and blocked...

omkar_savant 1y ago

Hey - I'm on the team that launched this. Please let me know if you have any questions!

SoKamil 1y ago
How are you going to deal with reCAPTCHA and ad impressions? Sounds like a conflict of interest.
- omkar_savant 1y ago
  No easy answers on this one unfortunately, lots of conversations ongoing on these - but our default stance has been to hand back control to the user in cases of captcha and have them solve these when they arise.
  - qingcharles 1y ago
    What about when all your competitors are solving the CAPTCHAs?
Awesomedonut 1y ago
Really cool stuff! Any interesting challenges the team ran into while developing it?
sumedh 1y ago
I am on https://gemini.browserbase.com/ https://gemini.browserbase.com/ and just click the use case mentioned on the site "Go to Hacker News and find the most controversial post from today, then read the top 3 comments and summarize the debate." It did not work, multiple times, just gets stuck after going to Hacker news.
bonoboTP 1y ago
It's a bit funny that I give Google Gemini a task and then it goes on the Google Search site and it gets stuck in the captcha tarpit that's supposed to block unwanted bots. But I guess Google Gemini shouldn't be unwanted for Google. Can't you ask the search team to whitelist the Gemini bot?

animanoir 1y ago

[dead]

martinald 1y ago

Interesting, seems to use 'pure' vision and x/y coords for clicking stuff. Most other browser automation with LLMs I've seen uses the dom/accessibility tree which absolutely churns through context, but is much more 'accurate' at clicking stuff because it can use the exact text/elements in a selector. Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.

deleted 1y ago
[deleted]
pbhjpbhj 1y ago
18 attempts - emulating the human HN experience when using mobile. Well, assuming it hit other links it didn't intend to anyway. /jk

John7878781 1y ago

[dead]

dekhn 1y ago

Many years ago I was sitting at a red light on a secondary road, where the primary cross road was idle. It seemed like you could solve this using a computer vision camera system that watched the primary road and when it was idle, would expedite the secondary road's green light. This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that can detect cars passing over - trivial hardware and software - and I concluded that my approach was just far too complicated and expensive. Similarly, when I look at computers, I typically want the ML/AI system to operate on a structured data that is codified for computer use. But I guess the world is complicated enough and computers got fast enough that having an AI look at a computer screen and move/click a mouse makes sense.

ge96 1y ago
It's funny I'll sometimes scoot forward/rock my car but I'm not sure if it's just coincidence. Also a lot of stop lights now have that tall white camera on top.
- bozhark 1y ago
  Like flashing lights for the first responders sensor
- Spooky23 1y ago
  Sometimes the rocking helps with a ground loop that isn’t working well.
- netghost 1y ago
  There's several mechanisms. The most common is (or at least was) a loop detector under the road that triggers when a vehicle is over it. Sometimes if you're not quite over it, or it's somewhat faulty that will trigger it.
trenchpilgrim 1y ago
FWIW those type of traffic cameras are in common use. https://www.milesight.com/company/blog/types-of-traffic-cameras https://www.milesight.com/company/blog/types-of-traffic-came...

AaronAPU 1y ago

I’m looking forward to a desktop OS optimized version so it can do the QA that I have no time for!

alexnewman 1y ago

A year ago I did something that used rag and accessibility mode to navigate ui.

realty_geek 1y ago

Absolutely hilarious how it gets stuck trying to solve captcha each time. I had to explicitly tell it not to go to google first. In the end I did manage to get it to play the housepriceguess game: https://www.youtube.com/watch?v=nqYLhGyBOnM https://www.youtube.com/watch?v=nqYLhGyBOnM I think I'll make that my equivalent of Simon Willison's "pelican riding a bicycle" test. It is fairly simple to explain but seems to trip up different LLMs in different ways.

dekhn 1y ago

I just have to say that I consider this an absolutely hilarious outcome. For many years, I focused on tech solutions that eliminated the need for a human to be in front of a computer doing tedious manual operations. For a wide range of activities, I proposed we focus on "turning everything in the world into database objects" so that computers could operate on them with minimal human effort. I spent significant effort in machine learning to achieve this. It didn't really occur to me that you could just train a computer to work directly on the semi-structured human world data (display screen buffer) through a human interface (mouse + keyboard). However, I fully support it (like all the other crazy ideas on the web that beat out the "theoretically better" approaches). I do not think it is unrealistic to expect that within a decade, we could have computer systems that can open chrome, start a video chat with somebody, go back and forth for a while to achieve a task, then hang up... with the person on the other end ever knowing they were dealing with a computer instead of a human.

TeMPOraL 1y ago
AI is succeeding where "theoretically better" approaches failed, because it addresses the underlying social problem. The computing ecosystem is an adversarial place, not a cooperative one. The reason we can't automate most of the tedium is by design - it's critical to how almost all money is made on the Internet. Can't monetize users when they automate your upsell channels and ad exposure away.
ncallaway 1y ago
> we could have computer systems that can open chrome, start a video chat with somebody, go back and forth for a while to achieve a task, then hang up... with the person on the other end ever knowing they were dealing with a computer instead of a human. Doesn't that...seem bad? I mean, it would certainly be a monumental and impressive technical accomplishment. But it still seems...quite bad to me.
- dekhn 1y ago
  Good or bad? I don't know. It just seems inevitable.

hipassage 1y ago

hi there

hipassage 1y ago

hi there, interesting post

deleted 1y ago

[deleted]

GeminiFan2025 1y ago

The new Gemini 2.5 model's ability to understand and interact with computer interfaces looks very impressive. It could be a game-changer for accessibility and automation. I wonder how robust it is with non-standard UI elements.

GeminiFan2025 1y ago

Impressive interface interaction by Gemini 2.5. Could be great for accessibility.

enjoylife 1y ago

> It is not yet optimized for desktop OS-level control Alas, AGI is not yet here. But I feel like if this OS-level of control was good enough, and the cost of the LLM in the loop wasn't bad, maybe that would be enough to kick start something akin to AGI.

alganet 1y ago
I am curious. Why do you think controlling an OS (and not just a browser) would be a move towards AGI?
- enjoylife 1y ago
  One thought is once it can touch the underlying system, it can provision resources, spawn processes, and persist itself, crossing the line from tool to autonomous entity. I admit you could do that in a browser shell nowadays, just maybe with more restrictions and guardrails. I don’t have any strong opinions here, but I do think a lower cost to escape the walled gardens agi starts in will be a factor
  - alganet 1y ago
    I see. I guarantee you that if AGI happens, it won't happen that way. No need to worry.
- throwaway-0001 1y ago
  Not just the os, but browsing control is enough to do 99% of the things he would want autonomously. Bank account + id + browser: has all the tools it needs to do many things: - earn money - allocate money - create accounts - delegate physical jobs to humans Create his own self loop in a server. Create a server account, use credit card + id provided, self host his own code… can now focus on getting more resources.
pseidemann 1y ago
Funny thing is, most humans cannot properly control a computer. Intelligence seems to be impossible to define.

mmaunder 1y ago

I prepare to be disappointed every time I click on a Google AI announcement. Which is so very unfortunate, given that they're the source of LLMs. Come on big G!! Get it together!

orliesaurus 1y ago

Does it know what's behind the "menu" of different apps? Or does it have to click on all menus and submenus to find out?

CryptoBanker 1y ago

Unless you give it specific instructions it will google something and give you the AI generated summary as the answer

mohsen1 1y ago

> Solve today's Wordle Stucks with: > ...the task is just to "solve today's Wordle", and as a web browsing robot, I cannot actually see the colors of the letters after a guess to make subsequent guesses. I can enter a word, but I cannot interpret the feedback (green, yellow, gray letters) to solve the puzzle.

Havoc 1y ago
So I guess it’s browsing in grey scale?
- egeozcan 1y ago
  I tested it and gemini seems not able to solve wordle in grayscale https://g.co/gemini/share/234fb68bc9a4 https://g.co/gemini/share/234fb68bc9a4
- daemonologist 1y ago
  It can definitely see color - I asked it to go to bing and search for the two most prominent colors in the bing background image and it did so just fine. It seems extremely lazy though; it prematurely reported as "completed" most of the tasks I gave it after the first or second step (navigating to the relevant website, usually).
  - avighnay 1y ago
    The models are mostly I believe capable of executing however as you rightly indicated 'lazy'. This 'laziness' I think is to conserve resource usage as much as possible as given the current state of AI market the infrastructure is being heavily subsidized for the user. This leads to perhaps the model being incentivized to produce an optimum result that satisfies the user by consuming the least amount of resources. This is also why most 'vibe' coding projects fail as the model is always going to give this optimum ('lazy') result by default. I have fun goading Gemini to break this ceiling when I work on my AI project - https://github.com/gingerhome/gingee https://github.com/gingerhome/gingee
jcims 1y ago

amelius 1y ago

Only use in environments where you can roll back everything.

sbinnee 1y ago

I think it’s related that I got an email from google, titled “ Simplifying your Gemini Apps experience”. It reads no privacy maximize AI. They are going to automatically collect data from all google apps, and users no longer have options to control access to individual apps.

zomgbbq 1y ago

I would love to use this for E2E testing. It would be great to make all my assertions with high level descriptions so tests are resilient to UI changes. Seems similar to the Amazon Nova Act API which is still in research preview.

sumedh 1y ago
Use Playwright and some AI model to write Playwright script, running those scripts will be much faster.
jbarber 1y ago
This is harder than you might expect because it's hard to tell whether a passing test is a false positive (i.e. the test passed, but it should have failed). It's also hard to convey to the testing system what is an acceptable level of change in the UI - what the testing system thinks is ok, you might consider broken. There are quite a few companies out there trying to solve this problem, including my previous employer https://rainforestqa.com https://rainforestqa.com
- nextworddev 1y ago
  LLM as judge

deleted 1y ago

[deleted]

Havoc 1y ago

At some point just having APIs for the web would just make sense. Rendering it and then throwing llms at interpreting it seems…suboptimal Impressive tech nonetheless

guelo 1y ago
The programmable web is a dead 20 year old dream that was crushed by the evil tech monopolists, Facebook, Google, etc. This emerging llm based automation tech is a glimmer of hope that we will be able to regain our data and autonomy.

keepamovin 1y ago

It’s basically an OODA loop. This is a good thing

btbuildem 1y ago

Does it work with ~legacy~ software? Eg, early 2000's Windows WhateverSoft's Widget Designer? Does it interface over COM? There's a goldmine to be had in automating ancient workflows that keep large corps alive.

tech234a 1y ago
“The Gemini 2.5 Computer Use model is primarily optimized for web browsers, but also demonstrates strong promise for mobile UI control tasks. It is not yet optimized for desktop OS-level control.”

jsrozner 1y ago

The irony is that most of tech companies make their money by forcing users to wade through garbage. For example, if you could browse the internet and avoid ads, why wouldn't you? If you could choose what twitter content to see outside of their useless algorithms, why wouldn't you?

machiaweliczny 1y ago
It's the same as saying if you could plunder and pillage why wouldn't you

tgsovlerkhgsel 1y ago

I'm so looking forward to it. Many of the problems that should be trivially solved with either AI or a script are hard to impossible to solve because the data is locked away in some form. Having an AI handle this may be inefficient, but as it uses the existing user interfaces, it might allow bypassing years of bureaucracy, and when the bureaucracy tries to fight it to justify its existence, it can fight it out with the EVERYONE MUST USE AI OR ELSE layers of management, while I can finally automate that idiotic task (using tens of kilowatts rather than a one-liner, but still better than having to do it by hand).

jwpapi 1y ago

Can somebody give me use cases that are faster than using an UX? How am I supposed to use this. I really can’t think of one, but I don’t want to be blind-sighted as obviously a lot of money is going into this. I also appreciate the tech behind it and functionality, but I still wonder for use cases

omkar_savant 1y ago
At current latency, there's a bunch of async automation usecases that one could use this for. For example: *Tedious to complete, easy to verify BPO: Filling out doctor licensing applications based on context from a web profile HR: Move candidate information from an ATS to HRIS system Logistics: Fill out a shipping order form based on a PDF of the packing label * Interact with a diverse set of sites for a single workflow Real estate: Diligence on properties that involves interacting with one of many county records websites Freight forwarding: Check the status of shipping containers across 1 of 50 port terminal sites Shipping: Post truck load requests across multiple job board sites BPO: Fetch the status of a Medicare coverage application from 1 of 50 state sites BPO: Fill out medical license forms across multiple state websites * Periodic syncs between various systems of record Clinical: Copy patient insurance info from Zocdoc into an internal system HR: Move candidate information from an ATS to HRIS system Customer onboarding: Create Salesforce tickets based on planned product installations that are logged in an internal system Logistics: Update the status of various shipments using tracking numbers on the USPS site * PDF extraction to system interaction Insurance: A broker processes a detailed project overview and creates a certificate of insurance with the specific details from the multi-page document by filling out an internal form Logistics: Fill out a shipping order form based on a PDF of the packing label Clinical: Enter patient appointment information into an EHR system based on a referral PDF Accounting: Extract invoice information from up to 50+ vendor formats and enter the details into a Google sheet without laborious OCR setup for specific formats Mortgage: Extract realtor names and address from a lease document and look up the license status on various state portals * Self healing broken RPA workflows
viking123 1y ago
Filling your CV in the workday applications

derekcheng08 1y ago

Really feels like computer use models may be vertical agent killers once they get good enough. Many knowledge work domains boil down to: use a web app, send an email. (e.g. recruiting, sales outreach)

loandbehold 1y ago
Why do you need an agent use web app through UI? Can't agent be integrated into web app natively? IMO for verticals you mentioned the missing piece is for an agent to be able to make phone calls.
- tgsovlerkhgsel 1y ago
  Native integration, APIs etc. require the web app author to do something. A computer use agent using the UI doesn't.

albert_e 1y ago

I believe it will need very capable but small VLMs that understand common User Interfaces very well -- small enough to run locally -- paired with any other higher level models on the cloud, to achieve human-speed interactions and beyond with reliability.

xrd 1y ago

I really want this model to try userinyerface.com

clayliu 1y ago

[dead]

krawcu 1y ago

I wonder how it would behave in a scenario where it has to download some file from a shady website that has all those advertisement with fake "download"

beepdyboop 1y ago
haha that's a great test actually

t43562 1y ago

How ironic that words become more powerful than images.....

skc 1y ago

How likely is it that the end game becomes that we stop writing apps for actual human users and instead sites become massive walls of minified text against a black screen.

barrenko 1y ago
Hopefully we get entirely off the internet.
fauigerzigerk 1y ago
If some functionality isn't used directly by humans why not expose it as an API? If you're asking how likely it is that all human-computer interaction will take place via lengthy natural language conversations then my guess is no. Visualising information and pointing at things is just too useful to replace it with what is essentially a smart command line interface.
password54321 1y ago
It is all just data. It doesn't need to be rendered to become input.
peytoncasper 1y ago
Actually a few startups working on this! You should check out Stytch isAgent SDK. We’re partnering with them on Web Bot Auth

SilverSlash 1y ago

Is this different from ChatGPT agent mode that I can use from the web app? I found that extremely useful for my task which required running some python and javascript code with open source libraries to generate an animated video effect. I greatly appreciated ChatGPT writing the code and then running it on OpenAI's VMs instead of me pasting that code on my machine. I wish Google released something like that in AI Studio.

nextworddev 1y ago
with this you can use your browser on device

ChaoPrayaWave 1y ago

I've always been interested in running LLM locally to automate browser tasks, but every time I've tried, I've found the browser API to be too complex. In contrast, writing scripts directly with Playwright or Puppeteer tends to be much more stable.

saltserv 1y ago

[dead]

nsonha 1y ago

Is there a claude code for computer use models? I mean something that's actually useful and not just a claude.ai kinda thing.

Michael_Keller 1y ago

[dead]

informal007 1y ago

Future will be more challenging for fraud-detection fields, good luck for them.

Muhvmd_4 1y ago

hfhgfdfgdgg