15 ms·
Web LLM – WebGPU Powered Inference of Large Language Models
- deleted 4y ago[deleted]
- brucethemoose2 4y ago> For example, the latest MacBook Pro can have more than 60G+ unified GPU RAM that can be used to store the model weights and a reasonably powerful GPU to run many workloads. ...for $3.5K minimum, according to the Apple website :/ Is there any chance WebGPU could utilize the matrix instructions shipping on newer/future IGPs? I think MLIR can do this through Vulkan, which is how SHARK is so fast in Stable Diffusion on the AMD 7900 series, but I know nothing about webgpu's restrictions or Apache TVM.
- summarity 4y agoThat would need support in dawn: https://dawn.googlesource.com/dawn https://dawn.googlesource.com/dawn Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.
- brucethemoose2 4y agoHmmm is there an issue tracker for Dawn matrix acceleration in... Vulkan or D3D12, I guess? Diving into these software stacks is making my head hurt.
- satvikpendem 4y agoGetting 64 GB of VRAM for $3.5k is a lot cheaper than buying the equivalent Nvidia discrete GPUs.
- summarity 4y agoIt's also interesting that this opens up the full saturation of Apple Silicon (minus the ANE): GGML can run on the CPU, using NEON and AMX, while another instance could run via Metal on the GPU using MLC/dawn. Though the two couldn't share (the same) memory at the moment.
- brucethemoose2 4y agoThe GPU's ML task energy is so much lower that you'd probably get better performance running everything on the GPU. I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
- j45 4y agoThis was a really insightful explanation, thanks. I have been wanting to get a beefier Mac Studio/mini m2 the more I’m seeing Apple Silicon specific tweaked packages.
- verdverm 4y agoYou can get it for a lot less from https://frame.work https://frame.work But 64G of VRAM is not the same as GPU mem, apples and oranges
- satvikpendem 4y agoWhere does Framework offer 64 GB of VRAM? By VRAM I am referring to GPU RAM, yes.
- brucethemoose2 4y agoTechnically any newish laptop with 64GB of RAM has 64GB of "VRAM," but right now the Apple M series and AMD 7000 series are the only IGPs with any significant ML power.
- 4y ago
- cahoot_bird 4y agoDo LLMs have a way around the high end GPU requirements, or can CPU code potentially be much more optimized somehow? This is the only thing I can think of, not everyone will have the latest high end GPUs to run such software..
- verdverm 4y agoCheck out LLAMA-CPP
- cahoot_bird 4y agoLooks like this was hacked together pretty quickly. This in CPU is exactly what needs optimized to run on more devices, if that's even possible.. I guess it will take hardware and software a while to catch up to compete with ChatGPT..
- verdverm 4y agoIf you look at the news, yes it came together quickly, but it has also gotten a lot of performance upgrades which have made significant improvements.
- m1el 4y agoIf you're doing inference on neural networks, each weight has to be read at least once per token. This means you're going to read at least the size of the entire model, per token, at least once during inference. If your model is 60GB, and you're reading it from the hard drive, then your bare minimum time of inference per token will be limited by your hard drive read throughput. Macbooks have ~4GB/s sequential read speed. Which means your inference time per token will be strictly more than 15 seconds. If your model is in RAM, then (according to Apple's advertising) your memory speed is 400GB/s, which is 100x your hard drive speed, and just the memory throughput will not be as much of a bottleneck here.
- rahimnathwani 4y agoYour answer applies equally to GPU and CPU, no? The comment to which you replied was asking about the need for a GPU, not the need for a lot of RAM.
- junrushao1994 4y agoTo clarify, running this WebLLM demo doesn't need a 3.5k MacBook Pro which costs $3.5k :-) WebGPU supports multiple backends, besides Metal on Apple Silicon, it offloads to Vulkan, DirectX, etc. It means a windows laptop with Vulkan support should work. My 2019 Intel MacBook with AMDGPU works as well. And of course, NVIDIA GPUs too! Our model is int4 quantized, and it is 4G in size, so it doesn't need 64GB memory either. Somewhere around 6G should suffice.
- bazmattaz 4y agoSorry non technical person here; has this been benchmarked against ChatGPT? Do you have any idea how it performs alongside GPT3 or GPT4?
- junrushao1994 4y agoThis is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.
- amelius 4y ago> This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( Compare using the loss function?
- junrushao1994 4y agoIn LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/ https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
- imranq 4y agoThere are plenty of LLM benchmarks that are used to test performance, some of them are: * Winogrande * BoolQ * PIQA * SIQA * HellaSwag etc...
- auggierose 4y agoJust tried out the demo, finally something that runs out of the box on my iMac Pro! This old 16GB card can finally breathe some LLM air!!
- bhouston 4y agoIt is funny how we have WebNN API coming, but it is so slow to arrive that we are misusing graphics APIs to do NN again. Anyone have a clue when WebNN API will arrive? It would be significantly more power efficient than using WebGPU/WebGL. https://www.w3.org/TR/webnn/ https://www.w3.org/TR/webnn/ Seems to be in development over at Chrome: https://chromestatus.com/feature/5738583487938560 https://chromestatus.com/feature/5738583487938560
- paulgb 4y agoI wouldn’t consider this to be “misusing” a graphics API in the same sense that tensor operations on top of WebGL are. WebGPU intentionally provides compute shaders with the goal of enabling non-graphical pipelines. > [WebGPU use cases include] Executing machine learning models efficiently on the GPU. It is possible to do general-purpose GPU (GPGPU) computation in WebGL, but it is sub-optimal and much more difficult. https://gpuweb.github.io/gpuweb/explainer/ https://gpuweb.github.io/gpuweb/explainer/
- rektide 4y agoI'm more worried that it'll end up kind of inferior to like, Google's other work, OpenXLA & iree, which seem to be the cross-target ML toolkit. https://news.ycombinator.com/item?id=35078410 https://news.ycombinator.com/item?id=35078410 I really want this to be good. It's hard to really trust the web spec is going to live up to what the other really really good web ML work is. But it'll at least unlock some good speed up, whenever it lands. I genuinely don't mind more delay, if it helps get things into a better position for long term wins.
- archerx 4y agoThis is really cool and I had no idea about it. I tried the live style transfer on live video demo and was amazed, even the with gpu polyfill I was getting 5fps on my underpowered laptop. It brought me back to the days of deepdream and deepstyle but live and on the web.
- hongkonger 4y agoYou guys are awesome. Both Web LLM and Web Stable Diffusion demos work on my Intel i3-1115G4 laptop with only 5.9GB of shared GPU memory.
- dwheeler 4y ago> Thanks to the open-source efforts like LLaMA ... Is the LLaMA model really open source now? Last I checked it was only licensed for non-commercial use, which isn't open source software at least. Have they changed the license? Are people depending on "databases can't be copyrighted"? Are people just presuming they won't be caught? There's lots of OSS that can use LLaMA but that's different from the model itself. This is a genuine question, people are making assertions but I can't find evidence for the assertions.
- Anduia 4y agoNot all implementations of it use the same license. See for example lit-llama by Lightning AI: https://news.ycombinator.com/item?id=35344787 https://news.ycombinator.com/item?id=35344787
- dwheeler 4y agoThat is for code to download the model, not for the model itself.
- jerpint 4y agoThey actually reimplemented the code to load the model. But it’s not clear if the weights of the model are free to use or not
- Tepix 3y agoWhat do you mean, it's not clear? https://github.com/facebookresearch/llama https://github.com/facebookresearch/llama links to https://forms.gle/jk851eBVbX1m5TAv5 https://forms.gle/jk851eBVbX1m5TAv5 which contains LLaMA license agreement below the form.
- dwheeler 3y agoOk, all evidence I can find is that the LLaMA model is not open source, and the article is just wrong. Software != model. If someone has evidence otherwise, please post.
- lxe 4y agoAh darn it, and here I was trying to do this with ggml and onxx.
- tantony 4y agoThe speed at which this stuff is developing makes it very hard to keep up and contribute in a meaningful way. Especially when you have a separate full time job.
- Hyzer 4y agoAre there any other projects/libraries that can run Llama models on Apple Silicon GPU? This is the first one I've seen. Comparing it to llama.cpp on my M1 Max 32GB, it seems at least as fast just by eyeballing it. Not sure if the inference speed numbers can be compared directly. vicuna-7b-v0 on Chrome Canary with the disable-robustness flag: encoding: 74.4460 tokens/sec, decoding: 18.0679 tokens/sec = 10.8ms per token llama.cpp: $ ./main -m models/7B/ggml-model-q4_0-ggjt.bin -t 8 --ignore-eos = 45 ms per token
- omneity 4y agoApple should really jump on this and support teams working on local LLMs.
- Tepix 3y agoHave you seen this? https://github.com/apple/ml-ane-transformers https://github.com/apple/ml-ane-transformers
- omneity 3y agoYes and it's definitely a step in the right direction. It's beyond me whether this is enough of an effort to support the current LLM movement. I expect Apple to be more proactive in these regards to capture the minds of a lightning-fast growing market increasingly slipping through its control (ChatGPT can be accessed from anywhere) and offer more incentives to developers, co-marketing being the lowest starting point.. probably something they're already working on and we didn't see the results of yet.
- j45 4y agoIt hard not to wonder if some of today llm use cases weren’t kept in mind on the current Apple Silicon roadmap.
- firatsarlar 4y agoIs this chasing impossible - not criticize, and love effort- ? But is it -a little- really possible to run an LLM in a single machine ? I want to believe :)
- ranguna 4y agoI don't get where your question is coming from, you can already run LLMs on a single machine. Checkout llama.cpp, tabby, text generation webui, gpt4all, AI Dungeon open source models like clover-edition, and know this we gpu based app.
- firatsarlar 3y agoThe question comes from a kind of confusion. We know the requirements of LLMs. How can we run the hardware it is currently working on, only the big LLM, with an 11Gb graphics card? I really didn't mind!
- quickthrower2 4y agoI just tried it and it works. And works amazing compared to anything that existed anywhere on earth one school term ago! so yeah why not?
- Tepix 3y agoGTPQ has been the missing piece, it allows quantizing the model weights from 16 to 4 bits with only a small loss in quality. That it turn allows running even the large 65 billion parameter version of the LLaMA model in ~33GB of RAM or VRAM. With VRAM that requires two 24GB GPUs which is no longer completely out of reach. The model running in the browser is a smaller version with 7 billion parameters, which is good enough for some things.
- amelius 4y agoOff topic. Did anyone use GPT to answer questions about datasheets of electronic components and how well did it work?
- quickthrower2 4y agoWorks well on my Dell XPS13 which is not particularly powerful but is modern. Answers are slower than chatGPT but not painfully slow and the response quality is great.
- tuukkah 3y agoWhat's the GPU in your model?
- xyproto 3y agoI gave WebLLM a spin. This is a promising start, but it's struggling with code (using vicuna-7b-v0): Write a Turbo Pascal program that says hello world. ```sql program helloWorld; begin write('H'); write('e'); write('l'); write('l'); writeln; end. ```