3 ms·
WebLLM: high-performance in-browser LLM inference engine
- refulgentis 1mo agoProject is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
- skybrian 1mo agoWhat did you switch to?
- seamossfet 1mo agoWe use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime
- theanonymousone 1mo agoTransformers.js supports a lot of models since v4
- refulgentis 1mo agoThere's, quietly, a llama.cpp WebGPU backend that works *great*. Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school. Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people on issues he's never seen their repro himself. Carefully pick models you can support down to "I can patch around the Jinja template engine". Go through the same llama web server APIs. c.f. github / telosnex / fllama if you need a reference (n.b. seeing peer comments, its much better than ONNX, ONNX never got within spitting distance of llama.cpp, my understanding from watching the LLM runtime is its used for Windows AI features so their models probably work great on Windows x Qualcomm. ONNX is great for smol models though, like VAD, a god send even.)
- roflcopter69 1mo agoI seem to be out of the loop when it comes to llama.cpp? What happened? Can you please provide some more pointers to sources so one can verify your claims? Do you see alternatives to llama.cpp in whose future you have more trust?
- woadwarrior01 1mo agoYeah, both WebLLM and mlc-llm have been mostly abandoned since NVIDIA acquired OctoAI in late 2024.
- TekMol 1mo agoThis seems to be the demo: https://chat.webllm.ai/ https://chat.webllm.ai/ I am getting: WebGPUNotAvailableError: WebGPU is not supported in your current environment, but it is necessary to run the WebLLM engine. On both, FireFox and Chromium on Linux.
- toto007 1mo agoYou can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
- gpugreg 1mo agoI tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://flags/#enable-webgpu-developer-features and I can now request a GPU with 'high-performance' power preference, although there is no way to check whether this actually selected the correct GPU, and it is useless to do it like that anyway, since no website actually sets this (experimental) option. Fortunately, there is the flag chrome://flags/#force-high-performance-gpu which should solve this, but it is not available for my platform. I have a fairly standard RTX 3060, but apparently, the most common GPU (according to the steam hardware survey) is not supported. Unfortunately, Firefox does not work any better. After setting the flag dom.webgpu.enabled to true in about:config, a few WebGPU examples (e.g. https://webgpu.github.io/webgpu-samples/?sample=helloTriangle https://webgpu.github.io/webgpu-samples/?sample=helloTriangl...) work, but many other examples crash the browser. And of course, Firefox can't select the correct GPU either.
- rzzzt 1mo agoPretty much the same here with Edge,#enable-unsafe-webgpu got me the emulated swiftshader adapter, but only #force-high-performance-gpu made the dGPU appear in the adapter list of https://webgpureport.org/ https://webgpureport.org/.
- nnevatie 1mo ago[flagged]
- asyncze 1mo ago[dead]
- ycCantCode 1mo ago[dead]
- conceptme 1mo agoBake me a cake responds with > Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.
- erichdongubler 1mo agoWhat browser and hardware are you using?
- adastra22 1mo agoA WebX technology that actually involves browsers!
- init0 1mo agoYou might like webml-kit https://npm.im/webml-kit https://npm.im/webml-kit
- MarioMan 1mo agoI really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.
- mandeepj 1mo agoIt is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.
- deleted 1mo ago[deleted]
- paidx 1mo ago[flagged]
- juggle73 1mo ago[dead]
- topaitools_xyz 1mo ago[flagged]
- LoganDark 1mo agoOh nice, this actually works in Safari on my Mac! Prefill: 181.2 tok/s, Decode: 58.5 tok/s on my M4 Max -- not too impressive for 1B, but definitely impressive that it runs in my browser!
- karuppiah7890 1mo agoI love it! It works fine for me on Google Chrome on my MacBook Pro 14-inch M5 16GB RAM and 10 Cores CPU and 10 Cores GPU, with Metal 4 support
- aitchnyu 28d agoIs there any use case/demo of lighter models like using embeddings for full text search, STT, OCR etc?