13 ms·
Show HN: WebGPU enables local LLM in the browser – demo site with AI chat
Browser LLM demo working on JavaScript and WebGPU.
WebGPU is already supported in Chrome, Safari, Firefox, iOS (v26) and Android.
Demo, similar to ChatGPT https://andreinwald.github.io/browser-llm/ https://andreinwald.github.io/browser-llm/
Code https://github.com/andreinwald/browser-llm https://github.com/andreinwald/browser-llm
- No need to use your OPENAI_API_KEY - its local model that runs on your device
- No network requests to any API
- No need to install any program
- No need to download files on your device (model is cached in browser)
- Site will ask before downloading large files (llm model) to browser cache
- Hosted on Github Pages from this repo - secure, because you see what you are running
- andsoitis 1y agovery cool. improvement would be if the input text box is always on screen, rather than having to manually scroll down as the screen fills.
- deleted 1y ago[deleted]
- maxmcd 1y agoLooks like this is a wrapper around: https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm Which has a full web demo: https://chat.webllm.ai/ https://chat.webllm.ai/
- refulgentis 1y agoI am to see it regardless - projects been very low activity for months. Just last night I was thinking about ripping it out before launch. No observable future. EDIT: Ah, wrapper ~= using it as a dependency, no more
- swores 1y agoIs this correct? It doesn't seem so to me, either from the way it works or from what little of the code I've looked at... But I don't have time to do more than the quick glance I just did at a few of the files of each and need to run, so hopefully someone cleverer than me who won't need as much time as me to answer the question could confirm while I'm afk
- refulgentis 1y agoEntirely correct, c.f. LLM.ts. The bit to Show HN here, is about ~40 lines of code. Simple Typescript MVP of calling the library. (source: maintains an LLM client that works across MLC/llama.cpp/3P providers; author of sibling comment that misunderstood initially)
- swores 1y agoThanks! (Have to admit your sibling comment confused the hell out of me.)
- pjmlp 1y agoBeware of opening this on mobile Internet.
- andreinwald 1y agoDemo site is asking before download
- lukan 1y agoWell, I am on a mobile right now, can someone maybe share anything about the performance?
- pjmlp 1y agoNot everyone enjoys unlimited data plans, and 500 MB is schon a lot.
- andreinwald 1y agoOn my Android device works pretty fast. But keep in mind that it's small Llama-3.2-1B model, specially for less powerfull GPU.
- asim 1y agoWhat's the performance of a model like vs an OpenAI API? What's the comparable here? Edit: I see it's same models locally that you'd run using Ollama or something else. So basically just constrained by the size of the model, GPU and perf of the machine.
- andreinwald 1y agoYes, its very similar to Ollama app, and Llama-3.2-1B model used
- petermcneeley 1y agoThis demo only works if you have the webgpu feature "f16". You can find out if you have this by checking for the feature list in https://webgpureport.org/ https://webgpureport.org/ . The page itself can of course check for this but since f16 support is common they probably just didnt bother.
- andreinwald 1y agoAlso here https://caniuse.com/webgpu https://caniuse.com/webgpu
- cgdl 1y agoWhich model does the demo use?
- solrith 1y ago[dead]
- andreinwald 1y agoLlama 3.2 1B
- scottfr 1y agoThere is a Prompt API in development that's available in both Chrome and Edge to give access to a local LLM. Chrome extensions have access to it and I believe websites can request access as part of an origin trial. The model is fully managed by the browser. It's currently the Gemini Nano model on Chrome, and they are testing a version of the Gemma 3n model in beta channels. Edge uses phi-4-mini. More information is available here: https://github.com/webmachinelearning/prompt-api https://github.com/webmachinelearning/prompt-api
- om8 1y agoTo have a gpu inference, you need a gpu. I have a demo that runs 8B llama on any computer with 4 gigs of ram https://galqiwi.github.io/aqlm-rs/about.html https://galqiwi.github.io/aqlm-rs/about.html
- echoangle 1y agoFun demo but the model that's used seems to be pretty stupid: > What's the best way to get to space? >> Unfortunately, it's not currently possible for humans to travel to space in the same way that astronauts do. While there have been several manned missions to space, such as those to the International Space Station, the technology and resources required to make interstellar travel feasible are still in the early stages of development. >> Currently, the fastest spacecraft is the Apollo missions, which took around 77 hours to reach the Moon. From there, they traveled to other planets, but the distances between them are so great that even at high speeds, it would take many years to reach them. >> As for traveling to space itself, there are no commercial or private companies that offer space travel services, and the cost of such a journey is estimated to be in the tens of millions of dollars per person. >> However, there are some concepts and technologies being explored that might one day make space travel more accessible and affordable. For example, private companies like SpaceX and Blue Origin are working on developing reusable rockets that could potentially take people to space more quickly and reliably. >> But for now, let's just say that space travel is still in the future, and we'll have to wait a bit longer for humans to venture into the cosmos!
- andreinwald 1y agoIt works on small Llama-3.2-1B model, specially for less powerfull GPU devices
- wongarsu 1y agoThe answer is still terrible for the model size. Maybe it's the 4 bit quantization, smaller models tend to react worse to that For reference, [1] is what stock quen3-0.6B would answer. Not a perfect answer, but much better at nearly half the number of parameters 1: https://markdownpastebin.com/?id=7ad4ad9f325d4354a858480abdc79268 https://markdownpastebin.com/?id=7ad4ad9f325d4354a858480abdc...
- KTibow 1y agoIt's likely the quantization on "Llama-3.2-1B-Instruct-q4f16_1-MLC". inference.net generated this more coherent answer: https://hst.sh/ovilewofox.md https://hst.sh/ovilewofox.md
- andreinwald 1y agoModel used: Llama 3.2 1B (small). Quality hould be similar with running Ollama app with same small model.
- Martin_Silenus 1y ago[dead]
- apitman 1y agoDoes anyone know why all of these WebGPU LLM demos have you download the models to browser storage rather than letting you open a gguf already on your local drive? I have several models downloaded already that I would be interested in trying.
- fooker 1y agoBrowsers are sandboxed away from user storage. You can change this by changing settings, command line arguments, build flags, etc. But can’t really expect people to do this just to use your website.
- apitman 1y agoYou can open a file for performant access in all major browsers. It's the same API used for uploading files (<input type="file" />), but you can also just load them into memory and do stuff.
- ethan_smith 1y agoWebGPU's security model restricts direct file system access, requiring models to be loaded through fetch/cache APIs rather than local file paths.
- cat-whisperer 1y agoI've been following the development of WebGPU and its potential applications, and this demo is a great example of what's possible.
- AndrewDucker 1y agoI asked "Why is the sky blue?" and got back a response of "coppia RR TalentDevExpressincer+'.//////////////////////////////////////////////////////////////////////// cha ولا.AutoSizesaving proleงคicate Like"/> infos эти za cornerback economical (%]\ enumligne.execRELEASEPropagation_station Bucks проHEME seas GASPOST[Unit(suffix Gloves" (and so on, for a few more paragraphs). Am I missing something?
- amelius 1y agoWell, at least it is more informative than an answer like "42".
- 0points 1y ago> Am I missing something? Realistic expectations.
- AndrewDucker 1y agoIt seems to be working for some people. I'm just curious whether there was something I could change to make it work.
- yahoozoo 1y agofuck me it’s hacking the internet
- gorkish 1y agoSet llm-auto-lobotomy to disabled in chrome:flags. I think this is only available in canary at the moment
- dataviz1000 1y agoVery cool! For some awesome examples of running different types of models in webgpu check out the examples folder here. https://github.com/huggingface/transformers.js https://github.com/huggingface/transformers.js
- gulan28 1y agoI did this with mlc @ https://wiz.chat https://wiz.chat some time ago. Warning: it has a llama 3.1 7b model and is around 4 gb. It needs either a GPU or a Macand works only on chrome
- RagnarD 1y agoCool idea but badly broken from a little testing.
- 201984 1y agoAny way to make this use my actual GPU instead of the terrible integrated one?
- countfeng 1y agoNow there is an error
- imatrix 1y agoIts not working. Loading model: waiting WebGPU error: WebGPU is NOT supported on this browser. Im on macOS and Safari.
- imatrix 1y agoSame on iOS and mobile Safari - not working.
- owebmaster 1y agotime to vote with your wallet, Safari does not support WebGPU.
- jsdeveloper 1y ago[dead]