4 ms·

Having played a LOT with browser use, playwright, and puppeteer (all via MCP integrations and pythonic test cases), it's incredibly clear how quickly Claude (in

Having played a LOT with browser use, playwright, and puppeteer (all via MCP integrations and pythonic test cases), it's incredibly clear how quickly Claude (in particular) loses the thread as it starts to interact with the browser. There's a TON of visual and contextual information that just vanishes as you begin to do anything particularly complex. In my experience, repeatedly forcing new context windows between screenshots has dramatically improved the ability for claude to perform complex intearctions in the browser, but it's all been pretty weak. When Claude can operate in the browser and effectively understand 5 radio buttons in a row, I think we'll have made real progress. So far, I've not seen that eval.

tripplyons 1y ago
Definitely a good idea to wait for real evidence of it working. Hopefully they aren't just using the same model that wasn't really trained for browser use.
MattSayar 1y ago
Same. When I try to get it to do a simple loop (eg take screenshot, click next, repeat) it'll work for about five iterations (out of a hundred or so desired) then say, "All done, boss!" I'm hoping Anthropic's browser extension is able to do some of the same "tricks" that Claude Code uses to gloss over these kinds of limitations.
- tripplyons 1y ago
  Hopefully one of those "tricks" involves training a model on examples of browser use.
- robots0only 1y ago
  Claude is extremely poor at vision when compared to Gemini and ChatGPT. i think anthropic severely overfit their evals to coding/text etc. use cases. maybe naively adding browser use would work, but I am a bit skeptical.
  - bdangubic 1y ago
    I have a completely different experience. Pasting a screenshot into CC is my de-facto go-to that more often than not leads to CC understanding what needs to be done etc…
    - deleted 1y ago
      [deleted]
  - user453 1y ago
    Is it overfitting if it makes them the best at those tasks?
- CSMastermind 1y ago
  This has been exactly my experience using all the browser based tools I've tried. ChatGPT's agents get the furthest but even then they only make it like 10 iterations or something.
  - rzzzt 1y ago
    I have better success with asking for a short script that does the million iterations than asking the thing to make the changes itself (edit: in IDEs, not in the browser).
    - seunosewa 1y ago
      If you need precision, that's the way to go, and it's usually cheaper and faster too.
- felarof 1y ago
  I'm wondering if they are using vanilla claude or if they are using a fine-tuned version of claude specifically for browser use. RL fine-tuning LLMs can have pretty amazing results. We did GRPO training of Qwen3:4B to do the task of a small action model at BrowserOS (https://www.browseros.com/ https://www.browseros.com/) and it was much better than running vanilla Claude, GPT.
philip1209 1y ago
Context rot: https://news.ycombinator.com/item?id=44564248 https://news.ycombinator.com/item?id=44564248
jascha_eng 1y ago
I have built a custom "deep research" internally that uses puppeteer to find business information, tech stack and other information about a company for our sales team. My experience was that giving the LLM a very limited set of tools and no screenshots worked pretty damn well. Tbf for my use case I don't need more interactivity than navigate_to_url and click_link. Each tool returning a text version of the page and the clickable options as an array. It is very capable of answering our basic questions. Although it is powered by gpt-5 not claude now.
- felarof 1y ago
  This is super cool! If a "deep research" like agent is available directly in your browser, would that be useful? We are building this at BrowserOS!
- panarky 1y ago
  Just shoving everything into one context fails after just a few turns. I've had more success with a hierarchy of agents. A supervisor agent stays focused on the main objective, and it has a plan to reach that objective that's revised after every turn. The supervisor agent invokes a sub-agent to search and select promising sites, and a separate sub-sub-agent for each site in the search results. When navigating a site that has many pages or steps, a sub-sub-sub-agent for each page or step can be useful. The sub-sub-sub-agent has all the context for that page or step, and it returns a very short summary of the content of that page, or the action it took on that step and the result to the sub-sub-agent. The sub-sub-agents return just the relevant details to their parent, the sub-agent. That way the supervisor agent can continue for many turns at the top level without exhausting the context window or losing the thread and pursuing its own objective.
  - jascha_eng 1y ago
    Hmm my browser agents each have about 50-100 turns (takes roughly 3-5 minutes for each one) and one focused objective I make use of structured output to group all the info it found into a standardized format at the end. I have 4 of those "research agents" with different prompts running after another and then I format the results into a nice slack message + Summarize and evaluate the results in one final call (with just the result jsons as input). This works really well. We use it to score leads as for how promising they are to reach out to for us.
rukuu001 1y ago
Maybe this will be the impetus for the ‘semantic web’ and accessibility to be taken seriously
suchintan 1y ago
Have you ever given Skyvern (https://github.com/Skyvern-AI/skyvern https://github.com/Skyvern-AI/skyvern) a try? I'd love to hear your opinion
lopis 1y ago
After all this time, we might be entering the age of proper web accessibility, because this will help AI helps understand pages better.

tripplyons 1y ago

Definitely a good idea to wait for real evidence of it working. Hopefully they aren't just using the same model that wasn't really trained for browser use.

MattSayar 1y ago

Same. When I try to get it to do a simple loop (eg take screenshot, click next, repeat) it'll work for about five iterations (out of a hundred or so desired) then say, "All done, boss!" I'm hoping Anthropic's browser extension is able to do some of the same "tricks" that Claude Code uses to gloss over these kinds of limitations.

tripplyons 1y ago
Hopefully one of those "tricks" involves training a model on examples of browser use.
robots0only 1y ago
Claude is extremely poor at vision when compared to Gemini and ChatGPT. i think anthropic severely overfit their evals to coding/text etc. use cases. maybe naively adding browser use would work, but I am a bit skeptical.
- bdangubic 1y ago
  I have a completely different experience. Pasting a screenshot into CC is my de-facto go-to that more often than not leads to CC understanding what needs to be done etc…
  - deleted 1y ago
    [deleted]
- user453 1y ago
  Is it overfitting if it makes them the best at those tasks?
CSMastermind 1y ago
This has been exactly my experience using all the browser based tools I've tried. ChatGPT's agents get the furthest but even then they only make it like 10 iterations or something.
- rzzzt 1y ago
  I have better success with asking for a short script that does the million iterations than asking the thing to make the changes itself (edit: in IDEs, not in the browser).
  - seunosewa 1y ago
    If you need precision, that's the way to go, and it's usually cheaper and faster too.
felarof 1y ago
I'm wondering if they are using vanilla claude or if they are using a fine-tuned version of claude specifically for browser use. RL fine-tuning LLMs can have pretty amazing results. We did GRPO training of Qwen3:4B to do the task of a small action model at BrowserOS (https://www.browseros.com/ https://www.browseros.com/) and it was much better than running vanilla Claude, GPT.

philip1209 1y ago

Context rot: https://news.ycombinator.com/item?id=44564248 https://news.ycombinator.com/item?id=44564248

jascha_eng 1y ago

I have built a custom "deep research" internally that uses puppeteer to find business information, tech stack and other information about a company for our sales team. My experience was that giving the LLM a very limited set of tools and no screenshots worked pretty damn well. Tbf for my use case I don't need more interactivity than navigate_to_url and click_link. Each tool returning a text version of the page and the clickable options as an array. It is very capable of answering our basic questions. Although it is powered by gpt-5 not claude now.

felarof 1y ago
This is super cool! If a "deep research" like agent is available directly in your browser, would that be useful? We are building this at BrowserOS!
panarky 1y ago
Just shoving everything into one context fails after just a few turns. I've had more success with a hierarchy of agents. A supervisor agent stays focused on the main objective, and it has a plan to reach that objective that's revised after every turn. The supervisor agent invokes a sub-agent to search and select promising sites, and a separate sub-sub-agent for each site in the search results. When navigating a site that has many pages or steps, a sub-sub-sub-agent for each page or step can be useful. The sub-sub-sub-agent has all the context for that page or step, and it returns a very short summary of the content of that page, or the action it took on that step and the result to the sub-sub-agent. The sub-sub-agents return just the relevant details to their parent, the sub-agent. That way the supervisor agent can continue for many turns at the top level without exhausting the context window or losing the thread and pursuing its own objective.
- jascha_eng 1y ago
  Hmm my browser agents each have about 50-100 turns (takes roughly 3-5 minutes for each one) and one focused objective I make use of structured output to group all the info it found into a standardized format at the end. I have 4 of those "research agents" with different prompts running after another and then I format the results into a nice slack message + Summarize and evaluate the results in one final call (with just the result jsons as input). This works really well. We use it to score leads as for how promising they are to reach out to for us.

rukuu001 1y ago

Maybe this will be the impetus for the ‘semantic web’ and accessibility to be taken seriously

suchintan 1y ago

Have you ever given Skyvern (https://github.com/Skyvern-AI/skyvern https://github.com/Skyvern-AI/skyvern) a try? I'd love to hear your opinion

lopis 1y ago

After all this time, we might be entering the age of proper web accessibility, because this will help AI helps understand pages better.