5 ms·

I built a very similar extension [1] a couple of months ago that supports a wide range of models, including Claude, and enables them to take control of a user's

I built a very similar extension [1] a couple of months ago that supports a wide range of models, including Claude, and enables them to take control of a user's browser using tools for mouse and keyboard actions, observation, etc. It's a fun little project to look at to understand how this type of thing works. It's clear to me that the tech just isn't there yet. The information density of a web page with standard representations (DOM, screenshot, etc) is an order of magnitude lower than that of, say, a document or piece of code, which is where LLMs shine. So we either need much better web page representations, or much more capable models, for this to work robustly. Having LLMs book flights by interacting with the DOM is sort of like having them code a web app using assembly. Dia, Comet, Browser Use, Gemini, etc are all attacking this and have big incentives to crack it, so we should expect decent progress here. A funny observation was that some models have been clearly fine tuned for web browsing tasks, as they have memorized specific selectors (e.g. "the selector for the search input in google search is `.gLFyf`"). [1] https://github.com/parsaghaffari/browserbee https://github.com/parsaghaffari/browserbee

akrymski 1y ago
I think this will fail for the same reason RSS failed - the business case just isn't there.
bboygravity 1y ago
I'm trying to build an automatic form filler (not just web-forms, any form) and I believe the secret lies in just chaining a whole bunch of LLM, OCR, form understanding and other API's together to get there. Just 1 LLM or agent is not going to cut it at the current state of art. Just looking at the DOM/clientside source doesn't work, because you're basically asking the LLM to act like a browser and redo the website rendering that the browser already does better (good luck with newer forms written in Angular bypassing the DOM). IMO the way to go is have the toolchain look at the forms/websites in the same way humans do (purely visually AFTER the rendering was done) and take it from there. Source: I tried to feed web source into LLMs and ask them to fill out forms (firefox addon), but webdevs are just too creative in the millions of ways they can ask for a simple freaking address (for example). Super tricky anyway, but there's no more annoying API than manually filling out forms, so worth the effort hopefully.
threatofrain 1y ago
> Having LLMs book flights by interacting with the DOM is sort of like having them code a web app using assembly. The DOM is merely inexpensive, but obviously the answer can't be solely in the DOM but in the visual representation layer because that's the final presentation to the user's face. Also the DOM is already the subject of cat and mouse games, this will just add a new scale and urgency to the problem. Now people will be putting fake content into the DOM and hiding content in the visual layer.
- jonplackett 1y ago
  It also surely leaves more room for prompt injection that the user can’t see
- mikepurvis 1y ago
  I had the same thought that really an LLM should interact with a browser viewport and just leverage normal accessibility features like tabbing between form fields and links, etc. Basically the LLM sees the viewport as a thumbnail image and goes “That looks like the central text, read that” and then some underlying skill implementation selects and returns the textual context from the viewport.
miguelspizza 1y ago
> It's clear to me that the tech just isn't there yet. Totally agree. This was the thesis behind MCP-B (now WebMCP https://github.com/MiguelsPizza/WebMCP https://github.com/MiguelsPizza/WebMCP) HN Post: https://news.ycombinator.com/item?id=44515403 https://news.ycombinator.com/item?id=44515403 DOM and visual parsing are dead ends for browser automation. Not saying models are bad; they are great. The web is just not designed for them at all. It's designed for humans, and humans, dare I say, are pretty impressive creatures. Providing an API contract between extensions and websites via MCP allows an AI to interact with a website as a first-class citizen. It just requires buy-in from website owners. It's being proposed as a web standard: > https://github.com/webmachinelearning/webmcp https://github.com/webmachinelearning/webmcp
- shermantanktop 1y ago
  > humans, dare I say, are pretty impressive creatures Damn straight. Humanism in the age of tech obsession seems to be contrarian. But when it takes billions of dollars to match a 5 year-old’s common sense, maybe we should be impressed by the 5 year old. They are amazing.
- chatmasta 1y ago
  I suspect this kind of framework will be adopted by websites with income streams that are not dependent on human attention (i.e. advertising revenue, mostly). They have no reason to resist LLM browser agents. But if they’re in the business of selling ads to human eyeballs, expect resistance. Maybe the AI companies will find a way to resell the user’s attention to the website, e.g. “you let us browse your site with an LLM, and we’ll show your ad to the user.”
  - onesociety2022 1y ago
    Even the websites whose primary source of revenue is not ad impressions might be resistant to let the agents be the primary interface through which users interact with their service. Instacart currently seems to be very happy to let ChatGPT Operator use its website to place an order (https://www.instacart.com/company/updates/ordering-groceries-with-an-ai-agent-like-openais-operator/ https://www.instacart.com/company/updates/ordering-groceries...) [1]. But what happens when the primary interface for shopping with Instacart is no longer their website or their mobile app? OpenAI could demand a huge take rate for orders placed via ChatGPT agents, and if they don't agree to it, ChatGPT can strike a deal with a rival company and push traffic to that service instead. I think Amazon is never going to agree to let other agents use its website for shopping for the same reason (they will restrict it to just Alexa). [1] - the funny part is the Instacart CEO quit shortly after this and joined OpenAI as CEO of Applications :)
asdff 1y ago
It is kind of funny how the systems are set up where there often is dense and queryable information out there already for a lot of these tasks, but these are ignored in favor of the difficult challenge of brute forcing the human consumer facing ui instead of some existing api that is designed to be machine readable already. E.g. booking flights. Travel agents use software that queries all the airlines ticket inventory to return flight information to you the consumer. The issue of booking a flight is theoretically solved already by virtue of these APIs that already exist to do just that. But for AI agents this is now a stumbling block because it would presumably take a little bit of time to craft out a rule to cover this edge case and return far more accurate information and results. Consumers with no alternative don't know what they are missing so there is no incentive to improve this.
- ambicapter 1y ago
  Those APIs aren't generally available to the public, are they?
  - asdff 1y ago
    Not always, but anthropic is not exactly the public either.
- dudeWithAMood 1y ago
  Dude you do not understand how bad those "APIs" are for booking flights. Customers of Travelport often have screen reading software that reads/writes to a green screen. There's also tele-type, but like most of the GDS providers use old IBM TPF mainframes. I spent the first two years of my career in the space, we joked anything invented post Michael Jackson's song Thriller wasn't present.
  - asdff 1y ago
    And yet, they exist, and software has been built on top of them already.
  - cicloid 1y ago
adam_arthur 1y ago
The LLM should not be seeing the raw DOM in its context window, but a highly simplified and compact version of it. In general LLMs perform worse both when the context is larger and also when the context is less information dense. To achieve good performance, all input to the prompt must be made as compact and information dense as possible. I built a similar tool as well, but for automating generation of E2E browser tests. Further, you can have sub-LLMs help with compacting aspects of the context prior to handing it off to the main LLM. (Note: it's important that, by design, HTML selectors cannot be hallucinated) Modern LLMs are absolutely capable of interpreting web pages proficiently if implemented well. That being said, things like this Claude product seem to be fundamentally poorly designed from both a security and general approach perspective and I don't agree at all that prompt engineering is remotely the right way to remediate this. There are so many companies pushing out junk products where the AI is just handling the wrong part of the loop and pulls in far too much context to perform well.
- antves 1y ago
  This is exactly it! We built a browser agent and got awesome results by designing the context in a simplified/compact version + using small/efficient LLMs - it's smooth.sh if you'd like to try
- felarof 1y ago
  > The LLM should not be seeing the raw DOM in its context window, but a highly simplified and compact version of it. Precisely! There is already something accessibility tree that Chromium rendering engine constructs which is a semantically meaningful version of the DOM. This is what we use at BrowserOS.com
- tempestn 1y ago
  Is it just me, or do both of my sibling comments pitching competing AI projects read like they're written by (the same underlying) AI?
  - bergie3000 1y ago
    You're exactly right! I see the problem now.
felarof 1y ago
Just dumping the raw DOM into the LLM context is brutal on token usage. We've seen pages that eat up 60-70k tokens when you include the full DOM plus screenshots, which basically maxes out your context window before you even start doing anything useful. We've been working on this exact problem at https://github.com/browseros-ai/BrowserOS https://github.com/browseros-ai/BrowserOS. Instead of throwing the entire DOM at the model, we hook into Chromium's rendering engine to extract a cleaner representation of what's actually on the page. Our browser agents work with this cleaned-up data, which makes the whole interaction much more efficient.
- commanderkeen08 1y ago
  Playwrights MCP went had a strong idea to default to the accessibility tree instead of DOM. Unfortunately, even that is pretty chonky.
- apitman 1y ago
  Maybe people will start making simpler/smaller websites in order to work better with AI tools. That would be nice.
  - pishpash 1y ago
    You just need to capture the rendering and represent that.
- edg5000 1y ago
  It could work simmilar to Claude Code right? Where it won't ingest the entire codebase, rather search for certain strings or start looking at a directed location and follow references from there. Indeed it seems infeasible to ingest the whole thing.
- kodefreeze 1y ago
  This is really interesting. We've been working on a smaller set of this problem space. We've also found in some cases you need to somehow pass to the model the sequence of events that happen (like a video of a transition). For instance, we were running a test case on a e commerce website and they have a random popup that used to come up after initial Dom was rendered but before action could be taken. This would confuse the LLM for the next action it needed to take because it didn't know the pop-up came up.
dotproto 1y ago
Just took a quick glance at your extension and observed that it's currently using the "debugger" permission. What features necessitated using this API rather than leveraging content scripts and less invasive WebExtensions APIs?
Exoristos 1y ago
Do we regret, yet, letting the Semantic Web wither on the vine?
- worthless-trash 1y ago
  /s no, because if it doesn't help people consume it is its NOT important.
- pishpash 1y ago
  You might get it when bots write pages.
- mike_hearn 1y ago
  It didn't really wither on the vine, it just moved to JSON REST APIs with React as the layer that maps the model to the view. What's missing is API discovery which MCP provides. The problem with the concept is not really the tech. The problem is the incentives. Companies don't have much incentive to offer APIs, in most cases. It just risks adding a middleman who will try and cut them out. Not many businesses want to be reduced to being just an API provider, it's a dead end business and thus a dead end career/lifestyle for the founders or executives. The telcos went through this in the early 2000s where their CEOs were all railing against a future of becoming "dumb pipes". They weren't able to stop it in the end, despite trying hard. But in many other cases companies did successfully avoid that fate. MCP+API might be different or it might not. It eliminates some of the downsides of classical API work like needing to guarantee stability and commit to a feature set. But it still poses the risk of losing control of your own brand and user experience. The obvious move is for OpenAI to come along and demand a rev share if too many customers are interacting with your service via ChatGPT, just like Google effectively demand a revshare for sending traffic to your website because so many customers interact with the internet via web search.
hinoki 1y ago
How do screen readers work? I’ve used all the aria- attributes to make automation/scraping hopefully more robust, but don’t have experience beyond that. Could accessibility attributes also help condense the content into something more manageable?
aminkhorrami 1y ago
Super cool

akrymski 1y ago

I think this will fail for the same reason RSS failed - the business case just isn't there.

bboygravity 1y ago

I'm trying to build an automatic form filler (not just web-forms, any form) and I believe the secret lies in just chaining a whole bunch of LLM, OCR, form understanding and other API's together to get there. Just 1 LLM or agent is not going to cut it at the current state of art. Just looking at the DOM/clientside source doesn't work, because you're basically asking the LLM to act like a browser and redo the website rendering that the browser already does better (good luck with newer forms written in Angular bypassing the DOM). IMO the way to go is have the toolchain look at the forms/websites in the same way humans do (purely visually AFTER the rendering was done) and take it from there. Source: I tried to feed web source into LLMs and ask them to fill out forms (firefox addon), but webdevs are just too creative in the millions of ways they can ask for a simple freaking address (for example). Super tricky anyway, but there's no more annoying API than manually filling out forms, so worth the effort hopefully.

threatofrain 1y ago

> Having LLMs book flights by interacting with the DOM is sort of like having them code a web app using assembly. The DOM is merely inexpensive, but obviously the answer can't be solely in the DOM but in the visual representation layer because that's the final presentation to the user's face. Also the DOM is already the subject of cat and mouse games, this will just add a new scale and urgency to the problem. Now people will be putting fake content into the DOM and hiding content in the visual layer.

jonplackett 1y ago
It also surely leaves more room for prompt injection that the user can’t see
mikepurvis 1y ago
I had the same thought that really an LLM should interact with a browser viewport and just leverage normal accessibility features like tabbing between form fields and links, etc. Basically the LLM sees the viewport as a thumbnail image and goes “That looks like the central text, read that” and then some underlying skill implementation selects and returns the textual context from the viewport.

miguelspizza 1y ago

> It's clear to me that the tech just isn't there yet. Totally agree. This was the thesis behind MCP-B (now WebMCP https://github.com/MiguelsPizza/WebMCP https://github.com/MiguelsPizza/WebMCP) HN Post: https://news.ycombinator.com/item?id=44515403 https://news.ycombinator.com/item?id=44515403 DOM and visual parsing are dead ends for browser automation. Not saying models are bad; they are great. The web is just not designed for them at all. It's designed for humans, and humans, dare I say, are pretty impressive creatures. Providing an API contract between extensions and websites via MCP allows an AI to interact with a website as a first-class citizen. It just requires buy-in from website owners. It's being proposed as a web standard: > https://github.com/webmachinelearning/webmcp https://github.com/webmachinelearning/webmcp

shermantanktop 1y ago
> humans, dare I say, are pretty impressive creatures Damn straight. Humanism in the age of tech obsession seems to be contrarian. But when it takes billions of dollars to match a 5 year-old’s common sense, maybe we should be impressed by the 5 year old. They are amazing.
chatmasta 1y ago
I suspect this kind of framework will be adopted by websites with income streams that are not dependent on human attention (i.e. advertising revenue, mostly). They have no reason to resist LLM browser agents. But if they’re in the business of selling ads to human eyeballs, expect resistance. Maybe the AI companies will find a way to resell the user’s attention to the website, e.g. “you let us browse your site with an LLM, and we’ll show your ad to the user.”
- onesociety2022 1y ago
  Even the websites whose primary source of revenue is not ad impressions might be resistant to let the agents be the primary interface through which users interact with their service. Instacart currently seems to be very happy to let ChatGPT Operator use its website to place an order (https://www.instacart.com/company/updates/ordering-groceries-with-an-ai-agent-like-openais-operator/ https://www.instacart.com/company/updates/ordering-groceries...) [1]. But what happens when the primary interface for shopping with Instacart is no longer their website or their mobile app? OpenAI could demand a huge take rate for orders placed via ChatGPT agents, and if they don't agree to it, ChatGPT can strike a deal with a rival company and push traffic to that service instead. I think Amazon is never going to agree to let other agents use its website for shopping for the same reason (they will restrict it to just Alexa). [1] - the funny part is the Instacart CEO quit shortly after this and joined OpenAI as CEO of Applications :)

asdff 1y ago

It is kind of funny how the systems are set up where there often is dense and queryable information out there already for a lot of these tasks, but these are ignored in favor of the difficult challenge of brute forcing the human consumer facing ui instead of some existing api that is designed to be machine readable already. E.g. booking flights. Travel agents use software that queries all the airlines ticket inventory to return flight information to you the consumer. The issue of booking a flight is theoretically solved already by virtue of these APIs that already exist to do just that. But for AI agents this is now a stumbling block because it would presumably take a little bit of time to craft out a rule to cover this edge case and return far more accurate information and results. Consumers with no alternative don't know what they are missing so there is no incentive to improve this.

ambicapter 1y ago
Those APIs aren't generally available to the public, are they?
- asdff 1y ago
  Not always, but anthropic is not exactly the public either.
dudeWithAMood 1y ago
Dude you do not understand how bad those "APIs" are for booking flights. Customers of Travelport often have screen reading software that reads/writes to a green screen. There's also tele-type, but like most of the GDS providers use old IBM TPF mainframes. I spent the first two years of my career in the space, we joked anything invented post Michael Jackson's song Thriller wasn't present.
- asdff 1y ago
  And yet, they exist, and software has been built on top of them already.
- cicloid 1y ago

adam_arthur 1y ago

The LLM should not be seeing the raw DOM in its context window, but a highly simplified and compact version of it. In general LLMs perform worse both when the context is larger and also when the context is less information dense. To achieve good performance, all input to the prompt must be made as compact and information dense as possible. I built a similar tool as well, but for automating generation of E2E browser tests. Further, you can have sub-LLMs help with compacting aspects of the context prior to handing it off to the main LLM. (Note: it's important that, by design, HTML selectors cannot be hallucinated) Modern LLMs are absolutely capable of interpreting web pages proficiently if implemented well. That being said, things like this Claude product seem to be fundamentally poorly designed from both a security and general approach perspective and I don't agree at all that prompt engineering is remotely the right way to remediate this. There are so many companies pushing out junk products where the AI is just handling the wrong part of the loop and pulls in far too much context to perform well.

antves 1y ago
This is exactly it! We built a browser agent and got awesome results by designing the context in a simplified/compact version + using small/efficient LLMs - it's smooth.sh if you'd like to try
felarof 1y ago
> The LLM should not be seeing the raw DOM in its context window, but a highly simplified and compact version of it. Precisely! There is already something accessibility tree that Chromium rendering engine constructs which is a semantically meaningful version of the DOM. This is what we use at BrowserOS.com
tempestn 1y ago
Is it just me, or do both of my sibling comments pitching competing AI projects read like they're written by (the same underlying) AI?
- bergie3000 1y ago
  You're exactly right! I see the problem now.

felarof 1y ago

Just dumping the raw DOM into the LLM context is brutal on token usage. We've seen pages that eat up 60-70k tokens when you include the full DOM plus screenshots, which basically maxes out your context window before you even start doing anything useful. We've been working on this exact problem at https://github.com/browseros-ai/BrowserOS https://github.com/browseros-ai/BrowserOS. Instead of throwing the entire DOM at the model, we hook into Chromium's rendering engine to extract a cleaner representation of what's actually on the page. Our browser agents work with this cleaned-up data, which makes the whole interaction much more efficient.

commanderkeen08 1y ago
Playwrights MCP went had a strong idea to default to the accessibility tree instead of DOM. Unfortunately, even that is pretty chonky.
apitman 1y ago
Maybe people will start making simpler/smaller websites in order to work better with AI tools. That would be nice.
- pishpash 1y ago
  You just need to capture the rendering and represent that.
edg5000 1y ago
It could work simmilar to Claude Code right? Where it won't ingest the entire codebase, rather search for certain strings or start looking at a directed location and follow references from there. Indeed it seems infeasible to ingest the whole thing.
kodefreeze 1y ago
This is really interesting. We've been working on a smaller set of this problem space. We've also found in some cases you need to somehow pass to the model the sequence of events that happen (like a video of a transition). For instance, we were running a test case on a e commerce website and they have a random popup that used to come up after initial Dom was rendered but before action could be taken. This would confuse the LLM for the next action it needed to take because it didn't know the pop-up came up.

dotproto 1y ago

Just took a quick glance at your extension and observed that it's currently using the "debugger" permission. What features necessitated using this API rather than leveraging content scripts and less invasive WebExtensions APIs?

Exoristos 1y ago

Do we regret, yet, letting the Semantic Web wither on the vine?

worthless-trash 1y ago
/s no, because if it doesn't help people consume it is its NOT important.
pishpash 1y ago
You might get it when bots write pages.
mike_hearn 1y ago
It didn't really wither on the vine, it just moved to JSON REST APIs with React as the layer that maps the model to the view. What's missing is API discovery which MCP provides. The problem with the concept is not really the tech. The problem is the incentives. Companies don't have much incentive to offer APIs, in most cases. It just risks adding a middleman who will try and cut them out. Not many businesses want to be reduced to being just an API provider, it's a dead end business and thus a dead end career/lifestyle for the founders or executives. The telcos went through this in the early 2000s where their CEOs were all railing against a future of becoming "dumb pipes". They weren't able to stop it in the end, despite trying hard. But in many other cases companies did successfully avoid that fate. MCP+API might be different or it might not. It eliminates some of the downsides of classical API work like needing to guarantee stability and commit to a feature set. But it still poses the risk of losing control of your own brand and user experience. The obvious move is for OpenAI to come along and demand a rev share if too many customers are interacting with your service via ChatGPT, just like Google effectively demand a revshare for sending traffic to your website because so many customers interact with the internet via web search.

hinoki 1y ago

How do screen readers work? I’ve used all the aria- attributes to make automation/scraping hopefully more robust, but don’t have experience beyond that. Could accessibility attributes also help condense the content into something more manageable?

aminkhorrami 1y ago

Super cool