4 ms·

I would prefer this as a browser extension, not as its own browser application.

by layer8 1y ago

felarof 1y ago
We would've preferred to build this as browser extension too. But we strongly believe that for building a good agent co-pilot we need bunch of changes at Chromium C++ code level. For example, chromium has a accessibility tree for every website, but doesn't expose it as an API to chrome extension. Having access to accessibility tree would greatly improve agent execution. We are also building bunch of changes in C++ for agents to interact with websites -- functions like click, elements with indexes. You can inject JS for doing this but it is 20-40X slower.
- esafak 1y ago
  Could you upstream that change in order to make it an extension in the future? I think people would not value it any less.
  - felarof 1y ago
    We don't mind upstreaming. But I don't think Google Chrome/Chromium wants to expose it as an API chrome extensions, if not they would've done this long time ago. From Google's perspective, extension are meant to be lightweight applications, with restricted access.
    - jazzyjackson 1y ago
      I'm not really interested in AI agents for my webbrowser, but it would be pretty cool to see a fork of chromium available that, aside from being de-googled, relaxes all the "restricted access" to make it more fun to modify and customize the way you guys are. Just a thought, may be more of a market for the framework more than the product :) See Sciter. A very cool, super lightweight alternative to Electron, but unfortunately it seems like a single developer project and I could never get any of the examples to run. https://sciter.com/ https://sciter.com/
      - felarof 1y ago
        Yes, we want to do this too! We'll expose much more richer APIs. What use-cases do you have in mind? like scraping?
      - Imustaskforhelp 1y ago
        I always wonder about what sort of js engine such projects use since at the end of the day imo, it is all just a dance b/w js engine, html and css. Html & Css feels a little solved problem but the problem is of the js engine. Sciter uses quickjs and I just checked and its like 35-36x times slower than V8 JIT Also another interesting rabbit hole is that I found Duktape in the quickjs benchmarks and I saw https://blogcpp.org/ https://blogcpp.org/ as one of the projects within Duktape but I can't even see the project on github. We really need some better way of preserving open source stuff I guess
        jazzyjackson 1y ago
        good callout wrt being slower than JIT. ofc for certain applications it's not a showstopper, ie, if you're not using javascript for your MVC but doing more of a progressive enhancement thing. CSS2 is closer to trivial, but CSS3 is practically a 3D game engine with all of its matrix transforms, transitions, animations, variables - not to mention all the different layout schemes (Sciter blogged about introducing display:flex and display:grid two months ago) The most interesting part of Sciter to me is that data persistence goes way beyond localStorage (string key: string value) or filesystem API, instead it's DyBase [0][1] behind the scenes, which looks to be a very intriguing style of storing trees of data in the host language's datatype (including whatever classes you define) without mucking about with the leaky abstractions of an ORM. [0] https://github.com/c-smile/sciter-js-sdk/blob/main/docs/md/storage/architecture.md https://github.com/c-smile/sciter-js-sdk/blob/main/docs/md/s... [1] http://www.garret.ru/dybase.html http://www.garret.ru/dybase.html
        mdaniel 1y ago
        > We really need some better way of preserving open source stuff I guess Not some rando's blog engine in C++, or other kinds of stupid throw-away code Anyway <https://web.archive.org/web/20241122030659/https://github.com/dertuxmalwieder/blogcpp?tab=readme-ov-file#first-things-first-how-can-i-get-it https://web.archive.org/web/20241122030659/https://github.co...> -> $(fossil clone https://code.rosaelefanten.org/blogcpp https://code.rosaelefanten.org/blogcpp) which takes a stunning amount of time but then reports Round-trips: 15 Artifacts sent: 0 received: 2751 Clone done, wire bytes sent: 4127 received: 124543126 remote: 2a03:4000:34:5e::1 Rebuilding repository meta-data... 100.0% complete... Extra delta compression... none found Vacuuming the database... project-id: 40a055cb170ae83c46b4ed9bf3b6a60e6e541aa0 server-id: cee9059305219c887fd29c677cbafb372252518a admin-user: mdaniel (password is "x2hEAaXUDj") opening the new ./blogcpp.fossil repository in directory ./blogcpp... 3rdparty/ConfigParser/ConfigParser.cpp ... project-name: BlogC++ repository: /home/runner/blogcpp.fossil project-code: 40a055cb170ae83c46b4ed9bf3b6a60e6e541aa0 checkout: fbea390316bc3aace7de0a9ccdba90ecc1949a10 2024-12-17 22:03:16 UTC parent: 9e604d205e7922ef1af87952fe1bebef0cbac336 2022-01-15 02:22:33 UTC tags: trunk comment: fuck you github (user: Cthulhux) check-ins: 20
- layer8 1y ago
  Would this be possible for Firefox?
  - felarof 1y ago
    IIRC, Firefox's web extension API does not provide access to accessibility tree as well.
    - dotancohen 1y ago
      I'm not GP, but I agree that if your goal is to empower the end user and protect him from corporate overlords, then Firefox is a more logical choice to fork from.
      - axus 1y ago
        I wonder if someone on the Chromium team will upstream all these BrowserOS changes, or "Not Invented Here" and re-implement it all for Gemini / Google Assistant. Can't imagine Firefox acquiring a Firefox fork!
        dotancohen 1y ago
        The hooks needed for integration will probably be implemented in Chrome, not Chromium. Nobody else should have them.
      - esperent 1y ago
        Isn't the Firefox code notoriously hard to fork and work with? I'm sure that nearly all of these Chrome forks would prefer to fork Firefox, but there's a reason they don't.
        layer8 1y ago
        The reason they usually don't is for better compatibility with more web pages.
- arjunchint 1y ago
  I mean you could build the agent with a first principles understanding of the DOM instead of just hacking together with the accessibility tree
- tunnelfx 1y ago
  How is that accessibility tree different from the “accessibility snapshot” that you can get from Playwright for example? I was tackling a similar problem few weeks ago and I found that playwright MCP was the most usable solution in my case. It doesn’t use an extension but it debugs the browser tabs (I guess using dev tools protocol) but I agree the experience was suboptimal
arjunchint 1y ago
We had this exact thought as well, you don't need a whole browser to implement the agentic capabilities, you can implement the whole thing with the limited permissions of a browser extension. There are plenty of zero day exploit patches that Google immediately rolls out and not to mention all the other features that Google doesn't push to Chromium. I wouldn't trust a random open source project for my day-to-day browser. Check out rtrvr.ai for a working implementation, we are an AI Web Agent browser extension that meets you where your workflows already are.
- felarof 1y ago
  Brave Browser (70M+ users) has validated that a chromium fork can be viable path. And it can in fact provide better privacy and security. Chrome extensions is not a bad idea too. Just saying that owning the underlying source code has some strong advantages in the long term (being able to use C++ for a11y tree, DOM handling, etc -- which will be 20-40X faster than injecting JS using chrome extension).
  - arjunchint 1y ago
    Honestly excited to see the benchmark result and comparison! Our benchmark results [https://www.rtrvr.ai/blog/web-bench-results https://www.rtrvr.ai/blog/web-bench-results] show that we are 7x faster than browser-use so curious to see if your claims live up to the hype
- esperent 1y ago
  > I wouldn't trust a random open source project for my day-to-day browser. Given that you're working on a direct competitor, this comment reads as fearmongering, designed to drive people over to your product.
  - arjunchint 1y ago
    I personally talked to another agentic browser player, fellou.ai, in the space asking them how they are keeping up with all the Chromium pushes as you need a dedicated team to handle the merges, they flat out told me they are targeting tech enthusiasts that are not interested in the security of their browser as much. As an ex-Google engineer I know the immense engineering efforts and infrastructure setup to develop Chrome. It is very implausible that two people can handle all the effort to serve a secure browser with 15+ million lines of constantly changing C++ code. A sandboxxed browser extension is the natural form factor for these agentic capabilities.
casslin 1y ago
try https://github.com/nanobrowser/nanobrowser https://github.com/nanobrowser/nanobrowser
Imustaskforhelp 1y ago
Exactly my thoughts when I saw nanobrowser being mentioned here.

I would prefer this as a browser extension, not as its own browser application.

by layer8 1y ago

I would prefer this as a browser extension, not as its own browser application.

We would've preferred to build this as browser extension too. But we strongly believe that for building a good agent co-pilot we need bunch of changes at Chromium C++ code level. For example, chromium has a accessibility tree for every website, but doesn't expose it as an API to chrome extension. Having access to accessibility tree would greatly improve agent execution. We are also building bunch of changes in C++ for agents to interact with websites -- functions like click, elements with indexes. You can inject JS for doing this but it is 20-40X slower.

esafak 1y ago
Could you upstream that change in order to make it an extension in the future? I think people would not value it any less.
- felarof 1y ago
  We don't mind upstreaming. But I don't think Google Chrome/Chromium wants to expose it as an API chrome extensions, if not they would've done this long time ago. From Google's perspective, extension are meant to be lightweight applications, with restricted access.
  - jazzyjackson 1y ago
    I'm not really interested in AI agents for my webbrowser, but it would be pretty cool to see a fork of chromium available that, aside from being de-googled, relaxes all the "restricted access" to make it more fun to modify and customize the way you guys are. Just a thought, may be more of a market for the framework more than the product :) See Sciter. A very cool, super lightweight alternative to Electron, but unfortunately it seems like a single developer project and I could never get any of the examples to run. https://sciter.com/ https://sciter.com/
    - felarof 1y ago
      Yes, we want to do this too! We'll expose much more richer APIs. What use-cases do you have in mind? like scraping?
    - Imustaskforhelp 1y ago
      I always wonder about what sort of js engine such projects use since at the end of the day imo, it is all just a dance b/w js engine, html and css. Html & Css feels a little solved problem but the problem is of the js engine. Sciter uses quickjs and I just checked and its like 35-36x times slower than V8 JIT Also another interesting rabbit hole is that I found Duktape in the quickjs benchmarks and I saw https://blogcpp.org/ https://blogcpp.org/ as one of the projects within Duktape but I can't even see the project on github. We really need some better way of preserving open source stuff I guess
      - jazzyjackson 1y ago
        good callout wrt being slower than JIT. ofc for certain applications it's not a showstopper, ie, if you're not using javascript for your MVC but doing more of a progressive enhancement thing. CSS2 is closer to trivial, but CSS3 is practically a 3D game engine with all of its matrix transforms, transitions, animations, variables - not to mention all the different layout schemes (Sciter blogged about introducing display:flex and display:grid two months ago) The most interesting part of Sciter to me is that data persistence goes way beyond localStorage (string key: string value) or filesystem API, instead it's DyBase [0][1] behind the scenes, which looks to be a very intriguing style of storing trees of data in the host language's datatype (including whatever classes you define) without mucking about with the leaky abstractions of an ORM. [0] https://github.com/c-smile/sciter-js-sdk/blob/main/docs/md/storage/architecture.md https://github.com/c-smile/sciter-js-sdk/blob/main/docs/md/s... [1] http://www.garret.ru/dybase.html http://www.garret.ru/dybase.html
      - mdaniel 1y ago
        > We really need some better way of preserving open source stuff I guess Not some rando's blog engine in C++, or other kinds of stupid throw-away code Anyway <https://web.archive.org/web/20241122030659/https://github.com/dertuxmalwieder/blogcpp?tab=readme-ov-file#first-things-first-how-can-i-get-it https://web.archive.org/web/20241122030659/https://github.co...> -> $(fossil clone https://code.rosaelefanten.org/blogcpp https://code.rosaelefanten.org/blogcpp) which takes a stunning amount of time but then reports Round-trips: 15 Artifacts sent: 0 received: 2751 Clone done, wire bytes sent: 4127 received: 124543126 remote: 2a03:4000:34:5e::1 Rebuilding repository meta-data... 100.0% complete... Extra delta compression... none found Vacuuming the database... project-id: 40a055cb170ae83c46b4ed9bf3b6a60e6e541aa0 server-id: cee9059305219c887fd29c677cbafb372252518a admin-user: mdaniel (password is "x2hEAaXUDj") opening the new ./blogcpp.fossil repository in directory ./blogcpp... 3rdparty/ConfigParser/ConfigParser.cpp ... project-name: BlogC++ repository: /home/runner/blogcpp.fossil project-code: 40a055cb170ae83c46b4ed9bf3b6a60e6e541aa0 checkout: fbea390316bc3aace7de0a9ccdba90ecc1949a10 2024-12-17 22:03:16 UTC parent: 9e604d205e7922ef1af87952fe1bebef0cbac336 2022-01-15 02:22:33 UTC tags: trunk comment: fuck you github (user: Cthulhux) check-ins: 20
layer8 1y ago
Would this be possible for Firefox?
- felarof 1y ago
  IIRC, Firefox's web extension API does not provide access to accessibility tree as well.
  - dotancohen 1y ago
    I'm not GP, but I agree that if your goal is to empower the end user and protect him from corporate overlords, then Firefox is a more logical choice to fork from.
    - axus 1y ago
      I wonder if someone on the Chromium team will upstream all these BrowserOS changes, or "Not Invented Here" and re-implement it all for Gemini / Google Assistant. Can't imagine Firefox acquiring a Firefox fork!
      - dotancohen 1y ago
        The hooks needed for integration will probably be implemented in Chrome, not Chromium. Nobody else should have them.
    - esperent 1y ago
      Isn't the Firefox code notoriously hard to fork and work with? I'm sure that nearly all of these Chrome forks would prefer to fork Firefox, but there's a reason they don't.
      - layer8 1y ago
        The reason they usually don't is for better compatibility with more web pages.
arjunchint 1y ago
I mean you could build the agent with a first principles understanding of the DOM instead of just hacking together with the accessibility tree
tunnelfx 1y ago
How is that accessibility tree different from the “accessibility snapshot” that you can get from Playwright for example? I was tackling a similar problem few weeks ago and I found that playwright MCP was the most usable solution in my case. It doesn’t use an extension but it debugs the browser tabs (I guess using dev tools protocol) but I agree the experience was suboptimal

arjunchint 1y ago

We had this exact thought as well, you don't need a whole browser to implement the agentic capabilities, you can implement the whole thing with the limited permissions of a browser extension. There are plenty of zero day exploit patches that Google immediately rolls out and not to mention all the other features that Google doesn't push to Chromium. I wouldn't trust a random open source project for my day-to-day browser. Check out rtrvr.ai for a working implementation, we are an AI Web Agent browser extension that meets you where your workflows already are.

felarof 1y ago
Brave Browser (70M+ users) has validated that a chromium fork can be viable path. And it can in fact provide better privacy and security. Chrome extensions is not a bad idea too. Just saying that owning the underlying source code has some strong advantages in the long term (being able to use C++ for a11y tree, DOM handling, etc -- which will be 20-40X faster than injecting JS using chrome extension).
- arjunchint 1y ago
  Honestly excited to see the benchmark result and comparison! Our benchmark results [https://www.rtrvr.ai/blog/web-bench-results https://www.rtrvr.ai/blog/web-bench-results] show that we are 7x faster than browser-use so curious to see if your claims live up to the hype
esperent 1y ago
> I wouldn't trust a random open source project for my day-to-day browser. Given that you're working on a direct competitor, this comment reads as fearmongering, designed to drive people over to your product.
- arjunchint 1y ago
  I personally talked to another agentic browser player, fellou.ai, in the space asking them how they are keeping up with all the Chromium pushes as you need a dedicated team to handle the merges, they flat out told me they are targeting tech enthusiasts that are not interested in the security of their browser as much. As an ex-Google engineer I know the immense engineering efforts and infrastructure setup to develop Chrome. It is very implausible that two people can handle all the effort to serve a secure browser with 15+ million lines of constantly changing C++ code. A sandboxxed browser extension is the natural form factor for these agentic capabilities.

casslin 1y ago

try https://github.com/nanobrowser/nanobrowser https://github.com/nanobrowser/nanobrowser

Imustaskforhelp 1y ago

Exactly my thoughts when I saw nanobrowser being mentioned here.