10 ms·

WASM Agents: AI agents running in the browser

169 points by selvan 1y ago

N_Lens 1y ago
I guess we're at the stage where every permutation of "AI Agents" and X (Where X is technologies & or spaces) must be tried and posted on HN.
- handfuloflight 1y ago
  That's either the peak of inflated expectations or the slope of enlightenment. Depending on what side you're on and "only time will tell."
- Dilettante_ 1y ago
  Sometimes you gotta shake the tree to see what falls out.
  - bravetraveler 1y ago
    Or pump the bubble
- selvan 1y ago
  Ship AI Agents as a web page :-)
- FrankyHollywood 1y ago
  We are in the early stage :) https://www.youtube.com/watch?v=gN-ZktmjIfE https://www.youtube.com/watch?v=gN-ZktmjIfE
  - latexr 1y ago
    That’s a well-remembered video, but I don’t really think it fits with the original comment. What we’re seeing is more akin to already having the plane flying (badly, still crashing frequently and landing in the wrong country) and instead of making it more reliable, everyone is trying different wheels and paint colours. We’re not really seeing any significant development with this. What LLMs need most desperately (and are far from getting) is reliability and not being convincing liars. Being able to query existing server models from your oven timer is a cool gimmick but not really transformative or advancing anything. It’s like a reverse JFK Space Effort speech: “We choose to indiscriminately throw at the wall every single LLM-adjacent idea we can think of. Not because it is useful, but because it is easy and potentially profitable”.
- spwa4 1y ago
  An excellent idea to have those be AI generated and posted! I'll start: https://chatgpt.com/share/68679564-6a44-8012-b1bd-25819bfbf0d9 https://chatgpt.com/share/68679564-6a44-8012-b1bd-25819bfbf0...
_joel 1y ago
Having to disable CORS restrictions is a bit meh, I understand why, but still.
raybb 1y ago
Can you bypass the cors issue with a browser extension? I seem to recall CORS doesn't apply to extensions, or at least the part that isn't injected to the webpages.
- ape4 1y ago
  CORS is mentioned on that page: NOTE: If you want to run tools that get information from some other server into your HTML page (e.g. the visit_webpage tool or the Ollama server itself), you need to make sure that CORS is enabled for those servers. For more information, refer to the troubleshooting section in our GitHub repository.
TekMol 1y ago
It seems the only code that runs in the browser here is the code that talks to LLMs on servers. Why would you need WASM for this?
- politelemon 1y ago
  They're using some python libraries like openai-agents so presumably it's to save on development efforts of calling/prompting/managing the HTTP endpoints. But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python.
  - yjftsjthsd-h 1y ago
    > But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python. Surely that's a prime use for AI?
- m13rar 1y ago
  From a quick gander. WASM is not to talk to the servers. WASM can be utilized to run AI Agents to talk to local LLMs from a sandboxed environment through the browser. For example in the next few years if Operating System companies and PC producers make small local models stock standards to improve the operating system functions and other services. This local LLM engine layer can be used by browser applications too and that being done through WASM without having to write Javascript and using WASM sandboxed layer to safely expose the this system LLM Engine Layer.
  - TekMol 1y ago
    No matter if the LLM is on the same machine or elsewhere, why would you need WASM to talk to it and not just JS?
- lgas 1y ago
  You never need WASM (or any other language, bytecode format, etc) to talk to LLMs. But WASM provides things people might like for agents, eg. strict sandboxing by default.
benatkin 1y ago
This is trying to use the word agent to make it sound cool, but it doesn't make a case for why it's particularly about agents and not just basic level AI stuff. > The agent code is nothing more than a Python script that relies on the openai-agents-python library to run an AI agent backed by an LLM served via an OpenAI-compatible API. The openai-agents-python code is useful for writing agents but it is possible to use it to write code that isn't very agentic. None of the examples are very agentic.
_l7dh 1y ago
Of course. Here is a corrected version of your text that fixes the grammar and typos while keeping the colloquial tone: I'd like to offer a less skeptical view on this, contrary to what I've read here so far. LLMs that act (a.k.a. agents) bring a whole lot of new security and privacy issues. If we were already heading toward a privacy dystopia (with trackers, centralized services, etc.), agents could take that to a whole new level. That's why I can only cheer when I see a different path where agents are run locally (by the way, Hugging Face has already published a couple of spaces demonstrating that). As a plus, because they're small, their environmental footprint will also be smaller (although, admittedly, I can also see the Jevons Paradox possibly happening here too).
sandGorgon 1y ago
i build an opensource mobile browser - we create ai agents (that run in the background) on the mobile browser. and build an extension framework on top so u can create these agents by publishing an extension. we hook into the android workmanager framework and do some quirky things with tab handling to make this work. its harder to do this on mobile than on desktop. bunch of people are trying to do interesting things like an automatic labubu purchase agent (on popmart) :D lots of purchase related agents pull requests welcome ! https://github.com/wootzapp/wootz-browser/pull/334 https://github.com/wootzapp/wootz-browser/pull/334
ipsum2 1y ago
I recently wrote some Javascript to automate clicking coupons. The website checks for non-human clicks using event.isTrusted. Firefox allowed me to bypass this by rewriting the JS to replace s/isTrusted/true, while Chrome Manifest V3 doesn't allow it. Anyway, Firefox might be the future of agents, due to its extensibility.
- deleted 1y ago
  [deleted]
_pdp_ 1y ago
Mildly interesting article - I mean, you can already run a ton of libraries that talk to an inference backend. The only difference here is that the client-side code is in Python, which by itself doesn't make creating agents any simpler - I would argue that it complicates things a tone. Also, connecting a model to a bunch of tools and dropping it into some kind of workflow is maybe 5% of the actual work. The rest is spent on observability, background tasks, queueing systems, multi-channel support for agents, user experience, etc., etc., etc. Nobody talks about that part, because most of the content out there is just chasing trends - without much real-world experience running these systems or putting them in front of actual customers with real needs.
- mentalgear 1y ago
  Agreed, regarding the other parts of the "LLM" stack, have a look at the, IMO best, LLM coordination / Observability platform TS library: https://mastra.ai/ https://mastra.ai/
  - _pdp_ 1y ago
    thanks, we are building chatbotkit.com / cbk.ai (not opensource)
meander_water 1y ago
When I saw the title, I thought this was running models in the browser. IMO that's way more interesting and you can do it with transformers.js and onnx runtime. You don't even need a gpu. https://huggingface.co/spaces/webml-community/llama-3.2-webgpu https://huggingface.co/spaces/webml-community/llama-3.2-webg...
- salviati 1y ago
  I think you _do_ need a GPU. But it can work with an integrated one, no need for a discrete one. I can't run it on Linux since WebGPU is not working for me...
  - pjmlp 1y ago
    It is yet not available on any browser on Linux, other than Android/Linux and ChromeOS.
zoobab 1y ago
No mention of WebGPU...
ultrathinkeng 1y ago
hmm
asim 1y ago
The frustrating thing about this is the limitation of using a browser. Agents should be long-running processes that exist external to a browser. The idea of using wasm is clever, but it feels like the entire browser environment needs to evolve because we're no longer dealing with just web pages. I think we are looking at a true evolution of the web now if this is the way it's going to go
- diggan 1y ago
  > Agents should be long-running processes that exist external to a browser Sure, but there are a ton of ways for doing that today. What this specific thing is addressing, is removing the requirement of "the dependency on extra tools and frameworks that need to be installed before the agents can be run".
- boomskats 1y ago
  That's what the Component Model[0] is all about. WASIp3[1] is gonna be awesome. Hopefully releasing later this year. [0]: https://component-model.bytecodealliance.org/ https://component-model.bytecodealliance.org/ [1]: https://wasi.dev/roadmap https://wasi.dev/roadmap
- simonw 1y ago
  When you say agents should be long running, which definition of "agent" are you talking about?
- evacchi 1y ago
  mcp.run is entirely based on wasm. Tools can run on our cloud or locally
dncornholio 1y ago
Putting Python code in a string in a html file is a big no no for me. We should be passed this. It looks like going 20 years back in time.
simonw 1y ago
In this case the "agent" definition they are using is the one from the https://github.com/openai/openai-agents-python https://github.com/openai/openai-agents-python Python library, which they are running in the browser via Pyodide and WASM. That library defines an agent as a system prompt and optional tools - notable because many other common agent definitions have the tools as required, not optional. That explains why their "hello world" demo just runs a single prompt: https://github.com/mozilla-ai/wasm-agents-blueprint/blob/main/demos/hello_agent.html https://github.com/mozilla-ai/wasm-agents-blueprint/blob/mai...
- lgas 1y ago
  > notable because many other common agent definitions have the tools as required, not optional. This feels weird to me. I would think of an agent with no tools as the trivial case of a "null agent" or "empty agent". It would be like saying you can't have a list with no elements because that's not a list at all... but an empty list is actually quite useful in many contexts. Which is why almost all implementations in all languages allow empty lists and add something distinct like a NonEmptyList to handle that specific case.
  - simonw 1y ago
    A lot of the agent/agentic definitions I see floating around are variants on "an LLM running tools in a loop". Can't do that without tools!
thepoet 1y ago
We looked at Pyodide and WASM along with other options like firecracker for our need of multi-step tasks that require running LLM generated code locally via Ollama etc. with some form of isolation than running it directly on our dev machines and figured it would be too much work with the various external libraries we have to install. The idea was to get code generated by a powerful remote LLM for general purpose stuff like video editing via ffmpeg, beautiful graphs generation via JS + chromium and stuff and execute it locally with all dependencies being installed before execution. We built CodeRunner (https://github.com/BandarLabs/coderunner https://github.com/BandarLabs/coderunner) on top of Apple Containers recently and have been using it for sometime. This works fine but still needs some improvement to work across very arbitrary prompts.
- indigodaddy 1y ago
  For the Gemini-cli integration, is the only difference between code runner with Gemini-cli, and gemini-cli itself, is that you are just using Gemini-cli in a container?
  - thepoet 1y ago
    No, Gemini-cli still is on your local machine, when it generates some code based on your prompt, with Coderunner, the code runs inside a container (which is inside a new lightweight VM courtesy Apple and provides VM level isolation), installs libraries requested, executes the generated code inside it and returns the result back to Gemini-cli. This is also not Gemini-cli specific and you could use the sandbox with any of the popular LLMs or even with your local ones.
    - indigodaddy 1y ago
      Thanks for explaining
    - indigodaddy 1y ago
      What would be the advantage of using your tool over say just using G CLI inside distrobox/toolbox/et al..?
om8 1y ago
I have a demo that runs llama3-{1,3,8}B in browser on cpu. It can be integrated with this thing in the future to be fully local https://galqiwi.github.io/aqlm-rs https://galqiwi.github.io/aqlm-rs

169 points by selvan 1y ago

I guess we're at the stage where every permutation of "AI Agents" and X (Where X is technologies & or spaces) must be tried and posted on HN.

handfuloflight 1y ago
That's either the peak of inflated expectations or the slope of enlightenment. Depending on what side you're on and "only time will tell."
Dilettante_ 1y ago
Sometimes you gotta shake the tree to see what falls out.
- bravetraveler 1y ago
  Or pump the bubble
selvan 1y ago
Ship AI Agents as a web page :-)
FrankyHollywood 1y ago
We are in the early stage :) https://www.youtube.com/watch?v=gN-ZktmjIfE https://www.youtube.com/watch?v=gN-ZktmjIfE
- latexr 1y ago
  That’s a well-remembered video, but I don’t really think it fits with the original comment. What we’re seeing is more akin to already having the plane flying (badly, still crashing frequently and landing in the wrong country) and instead of making it more reliable, everyone is trying different wheels and paint colours. We’re not really seeing any significant development with this. What LLMs need most desperately (and are far from getting) is reliability and not being convincing liars. Being able to query existing server models from your oven timer is a cool gimmick but not really transformative or advancing anything. It’s like a reverse JFK Space Effort speech: “We choose to indiscriminately throw at the wall every single LLM-adjacent idea we can think of. Not because it is useful, but because it is easy and potentially profitable”.
spwa4 1y ago
An excellent idea to have those be AI generated and posted! I'll start: https://chatgpt.com/share/68679564-6a44-8012-b1bd-25819bfbf0d9 https://chatgpt.com/share/68679564-6a44-8012-b1bd-25819bfbf0...

_joel 1y ago

Having to disable CORS restrictions is a bit meh, I understand why, but still.

raybb 1y ago

Can you bypass the cors issue with a browser extension? I seem to recall CORS doesn't apply to extensions, or at least the part that isn't injected to the webpages.

ape4 1y ago
CORS is mentioned on that page: NOTE: If you want to run tools that get information from some other server into your HTML page (e.g. the visit_webpage tool or the Ollama server itself), you need to make sure that CORS is enabled for those servers. For more information, refer to the troubleshooting section in our GitHub repository.

TekMol 1y ago

It seems the only code that runs in the browser here is the code that talks to LLMs on servers. Why would you need WASM for this?

politelemon 1y ago
They're using some python libraries like openai-agents so presumably it's to save on development efforts of calling/prompting/managing the HTTP endpoints. But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python.
- yjftsjthsd-h 1y ago
  > But yes this could just be done in regular JS in the browser, they'd have to write a lot of boilerplate for an ecosystem which is mainly Python. Surely that's a prime use for AI?
m13rar 1y ago
From a quick gander. WASM is not to talk to the servers. WASM can be utilized to run AI Agents to talk to local LLMs from a sandboxed environment through the browser. For example in the next few years if Operating System companies and PC producers make small local models stock standards to improve the operating system functions and other services. This local LLM engine layer can be used by browser applications too and that being done through WASM without having to write Javascript and using WASM sandboxed layer to safely expose the this system LLM Engine Layer.
- TekMol 1y ago
  No matter if the LLM is on the same machine or elsewhere, why would you need WASM to talk to it and not just JS?
lgas 1y ago
You never need WASM (or any other language, bytecode format, etc) to talk to LLMs. But WASM provides things people might like for agents, eg. strict sandboxing by default.

benatkin 1y ago

This is trying to use the word agent to make it sound cool, but it doesn't make a case for why it's particularly about agents and not just basic level AI stuff. > The agent code is nothing more than a Python script that relies on the openai-agents-python library to run an AI agent backed by an LLM served via an OpenAI-compatible API. The openai-agents-python code is useful for writing agents but it is possible to use it to write code that isn't very agentic. None of the examples are very agentic.

_l7dh 1y ago

Of course. Here is a corrected version of your text that fixes the grammar and typos while keeping the colloquial tone: I'd like to offer a less skeptical view on this, contrary to what I've read here so far. LLMs that act (a.k.a. agents) bring a whole lot of new security and privacy issues. If we were already heading toward a privacy dystopia (with trackers, centralized services, etc.), agents could take that to a whole new level. That's why I can only cheer when I see a different path where agents are run locally (by the way, Hugging Face has already published a couple of spaces demonstrating that). As a plus, because they're small, their environmental footprint will also be smaller (although, admittedly, I can also see the Jevons Paradox possibly happening here too).

sandGorgon 1y ago

i build an opensource mobile browser - we create ai agents (that run in the background) on the mobile browser. and build an extension framework on top so u can create these agents by publishing an extension. we hook into the android workmanager framework and do some quirky things with tab handling to make this work. its harder to do this on mobile than on desktop. bunch of people are trying to do interesting things like an automatic labubu purchase agent (on popmart) :D lots of purchase related agents pull requests welcome ! https://github.com/wootzapp/wootz-browser/pull/334 https://github.com/wootzapp/wootz-browser/pull/334

ipsum2 1y ago

I recently wrote some Javascript to automate clicking coupons. The website checks for non-human clicks using event.isTrusted. Firefox allowed me to bypass this by rewriting the JS to replace s/isTrusted/true, while Chrome Manifest V3 doesn't allow it. Anyway, Firefox might be the future of agents, due to its extensibility.

deleted 1y ago
[deleted]

_pdp_ 1y ago

Mildly interesting article - I mean, you can already run a ton of libraries that talk to an inference backend. The only difference here is that the client-side code is in Python, which by itself doesn't make creating agents any simpler - I would argue that it complicates things a tone. Also, connecting a model to a bunch of tools and dropping it into some kind of workflow is maybe 5% of the actual work. The rest is spent on observability, background tasks, queueing systems, multi-channel support for agents, user experience, etc., etc., etc. Nobody talks about that part, because most of the content out there is just chasing trends - without much real-world experience running these systems or putting them in front of actual customers with real needs.

mentalgear 1y ago
Agreed, regarding the other parts of the "LLM" stack, have a look at the, IMO best, LLM coordination / Observability platform TS library: https://mastra.ai/ https://mastra.ai/
- _pdp_ 1y ago
  thanks, we are building chatbotkit.com / cbk.ai (not opensource)

meander_water 1y ago

When I saw the title, I thought this was running models in the browser. IMO that's way more interesting and you can do it with transformers.js and onnx runtime. You don't even need a gpu. https://huggingface.co/spaces/webml-community/llama-3.2-webgpu https://huggingface.co/spaces/webml-community/llama-3.2-webg...

salviati 1y ago
I think you _do_ need a GPU. But it can work with an integrated one, no need for a discrete one. I can't run it on Linux since WebGPU is not working for me...
- pjmlp 1y ago
  It is yet not available on any browser on Linux, other than Android/Linux and ChromeOS.

zoobab 1y ago

No mention of WebGPU...

ultrathinkeng 1y ago

hmm

asim 1y ago

The frustrating thing about this is the limitation of using a browser. Agents should be long-running processes that exist external to a browser. The idea of using wasm is clever, but it feels like the entire browser environment needs to evolve because we're no longer dealing with just web pages. I think we are looking at a true evolution of the web now if this is the way it's going to go

diggan 1y ago
> Agents should be long-running processes that exist external to a browser Sure, but there are a ton of ways for doing that today. What this specific thing is addressing, is removing the requirement of "the dependency on extra tools and frameworks that need to be installed before the agents can be run".
boomskats 1y ago
That's what the Component Model[0] is all about. WASIp3[1] is gonna be awesome. Hopefully releasing later this year. [0]: https://component-model.bytecodealliance.org/ https://component-model.bytecodealliance.org/ [1]: https://wasi.dev/roadmap https://wasi.dev/roadmap
simonw 1y ago
When you say agents should be long running, which definition of "agent" are you talking about?
evacchi 1y ago
mcp.run is entirely based on wasm. Tools can run on our cloud or locally

dncornholio 1y ago

Putting Python code in a string in a html file is a big no no for me. We should be passed this. It looks like going 20 years back in time.

simonw 1y ago

In this case the "agent" definition they are using is the one from the https://github.com/openai/openai-agents-python https://github.com/openai/openai-agents-python Python library, which they are running in the browser via Pyodide and WASM. That library defines an agent as a system prompt and optional tools - notable because many other common agent definitions have the tools as required, not optional. That explains why their "hello world" demo just runs a single prompt: https://github.com/mozilla-ai/wasm-agents-blueprint/blob/main/demos/hello_agent.html https://github.com/mozilla-ai/wasm-agents-blueprint/blob/mai...

lgas 1y ago
> notable because many other common agent definitions have the tools as required, not optional. This feels weird to me. I would think of an agent with no tools as the trivial case of a "null agent" or "empty agent". It would be like saying you can't have a list with no elements because that's not a list at all... but an empty list is actually quite useful in many contexts. Which is why almost all implementations in all languages allow empty lists and add something distinct like a NonEmptyList to handle that specific case.
- simonw 1y ago
  A lot of the agent/agentic definitions I see floating around are variants on "an LLM running tools in a loop". Can't do that without tools!

thepoet 1y ago

We looked at Pyodide and WASM along with other options like firecracker for our need of multi-step tasks that require running LLM generated code locally via Ollama etc. with some form of isolation than running it directly on our dev machines and figured it would be too much work with the various external libraries we have to install. The idea was to get code generated by a powerful remote LLM for general purpose stuff like video editing via ffmpeg, beautiful graphs generation via JS + chromium and stuff and execute it locally with all dependencies being installed before execution. We built CodeRunner (https://github.com/BandarLabs/coderunner https://github.com/BandarLabs/coderunner) on top of Apple Containers recently and have been using it for sometime. This works fine but still needs some improvement to work across very arbitrary prompts.

indigodaddy 1y ago
For the Gemini-cli integration, is the only difference between code runner with Gemini-cli, and gemini-cli itself, is that you are just using Gemini-cli in a container?
- thepoet 1y ago
  No, Gemini-cli still is on your local machine, when it generates some code based on your prompt, with Coderunner, the code runs inside a container (which is inside a new lightweight VM courtesy Apple and provides VM level isolation), installs libraries requested, executes the generated code inside it and returns the result back to Gemini-cli. This is also not Gemini-cli specific and you could use the sandbox with any of the popular LLMs or even with your local ones.
  - indigodaddy 1y ago
    Thanks for explaining
  - indigodaddy 1y ago
    What would be the advantage of using your tool over say just using G CLI inside distrobox/toolbox/et al..?

om8 1y ago

I have a demo that runs llama3-{1,3,8}B in browser on cpu. It can be integrated with this thing in the future to be fully local https://galqiwi.github.io/aqlm-rs https://galqiwi.github.io/aqlm-rs