6 ms·

To clarify, running this WebLLM demo doesn't need a 3.5k MacBook Pro which costs $3.5k :-) WebGPU supports multiple backends, besides Metal on Apple Silicon, i

by junrushao1994 3y ago

To clarify, running this WebLLM demo doesn't need a 3.5k MacBook Pro which costs $3.5k :-) WebGPU supports multiple backends, besides Metal on Apple Silicon, it offloads to Vulkan, DirectX, etc. It means a windows laptop with Vulkan support should work. My 2019 Intel MacBook with AMDGPU works as well. And of course, NVIDIA GPUs too! Our model is int4 quantized, and it is 4G in size, so it doesn't need 64GB memory either. Somewhere around 6G should suffice.

bazmattaz 3y ago
Sorry non technical person here; has this been benchmarked against ChatGPT? Do you have any idea how it performs alongside GPT3 or GPT4?
- junrushao1994 3y ago
  This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.
  - amelius 3y ago
    > This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( Compare using the loss function?
    - junrushao1994 3y ago
      In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/ https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
  - imranq 3y ago
    There are plenty of LLM benchmarks that are used to test performance, some of them are: * Winogrande * BoolQ * PIQA * SIQA * HellaSwag etc...
    - junrushao1994 3y ago
      Would be nice if anyone could help us benchmark! Our primary focus though is not model performance, but to demonstrate the capability that TVM Unity generates code targeting WebGPU and allows them to run with client GPUs :-)
- wokwokwok 3y ago
  It’s rubbish compared to GPT-3/4. Any model you can trivially load in your browser will be significantly smaller than those models, and broadly speaking smaller = worse. This example is a 4 GB model, that’s (I guess) based off some smallish model like the llama 7B. It’s a proof of concept, not a chat-gpt replacement. There’s nothing here that’s new other than “runs in the browser”; so it won’t be better than any other model you can get your hands on. This kind of thing should be label ByoM (bring your own model). The model isn’t the interesting part of this.
  - junrushao1994 3y ago
    To share some fun stuff, here is the response generated by this model: As an AI language model, I would respond by acknowledging that the model discussed in the message is indeed smaller than some of the larger language models like GPT-3/4. However, I would also point out that size is not always a direct indicator of performance, and that the model may still be able to perform well on certain tasks or in certain contexts. Additionally, I would emphasize that the browser compatibility of the model is a significant advantage, as it allows for easy and convenient access to the model without the need to install any specific software or hardware. Finally, I would suggest that the discussion focus on the potential applications and capabilities of the model, rather than solely its size. Back to the topic, we don't make much innovation on the model, so I am probably not the best person to evaluate how a model compares with SOTAs. There are indeed lots of super cool techniques being explored lately that makes it possible to deploy smaller and smaller models, for example, LLM.int8() [1] and int4 quantization [2] without loss of zero-shot accuracy. Can't predict the future, but maybe one day there will be something really powerful but small enough to fit in the pocket of everyone :-) [1] Dettmers, Tim, et al. "LLM. int8 (): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022). [2] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit Inference Scaling Laws." arXiv preprint arXiv:2212.09720 (2022).
    - wokwokwok 3y ago
      > but maybe one day There you go, summarised for you. You can hand wave about quantised models til the end of time but specifically this model is a trivial toy model. No amount of pondering about the future avoid the fundamental fact that small models (~7B) are inferior to larger models like GPT. It’s dishonest to suggest otherwise. :( There’s no reason to do this other than selling snake oil. Maybe. One day. In the future. Things might be different. right now they are not.
- jimmySixDOF 3y ago
  I'm also not sure what the real use case is here other than an intellectual experiment. I may be missing the point but as I understand the world, running a Local LLM has nothing to do with the web in any way while it is doing so and so what's the benefit of WebGPU ? If you are streaming some constantly changing live data 3D experience in real time served over the internet by a website you just clicked on then WebGPU is making life easier. If you have a local pile of training data and weights sitting there on your hard drive for serious crunching then its like tying your shoelaces together just because developers want to have write once run everywhere dx which is nice for them but value sould always favor the user. Happy for someone to change my mind.
  - bigdict 3y ago
    WebGPU is much more portable than CUDA.
slimsag 3y ago
Nice work! I knew it wouldn't be long before someone put this together :) I'm curious if given a different language (like Zig) with WebGPU access if you could easily translate that last-mile of code to execute there or not? In specific I wonder if I can do it, and if you can give me an overview of where the code for "Universal deployment" in your diagram actually lives? I found llm_chat.js, but it seems that doesn't include the logic necessary for building WGSL shaders? Am I wrong or does that happen elsewhere like in the TVM runtime? How much is baked into llm_chat.wasm, and where is the source for that?
- crowwork 3y ago
  The WGSL are generated and compiled through TVM and embedded into the wasm. I think what you mean is wgpu native support. At the moment the web gpu runtime dispatches to the js webgpu environment. Once TVM runtime comes with wgpu native support (like the current ones in vulkan or metal), then it is possible to leverage any wgpu native runtime like what Zig provide. Additionally, currently tvm natively support targets like vulkan, metal directly which allows targeting these other platforms
  - slimsag 3y ago
    OK that makes sense; so basically if I want to give this a shot then I would just need to read llm_chat.js and the TVM docs, and translate llm_chat.js to my language of choice effectively?
    - crowwork 3y ago
      I think instead what would be needed is a wgpu native runtime support for TVM. Like the implementations in tvm vulkan, then it will be naturally link to any runtime that provides webgpu.h Then yah the llm_chat.js would be high-level logic that targets the tvm runtime, and can be implemented in any language that tvm runtime support(that includes, js, java, c++ rust etc). Support webgpu native is an interesting direction. Feel free to open a thread in tvm discuss forum and perhaps there would be fun things to collaborate in OSS

bazmattaz 3y ago

Sorry non technical person here; has this been benchmarked against ChatGPT? Do you have any idea how it performs alongside GPT3 or GPT4?

junrushao1994 3y ago
This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.
- amelius 3y ago
  > This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-( Compare using the loss function?
  - junrushao1994 3y ago
    In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/ https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
- imranq 3y ago
  There are plenty of LLM benchmarks that are used to test performance, some of them are: * Winogrande * BoolQ * PIQA * SIQA * HellaSwag etc...
  - junrushao1994 3y ago
    Would be nice if anyone could help us benchmark! Our primary focus though is not model performance, but to demonstrate the capability that TVM Unity generates code targeting WebGPU and allows them to run with client GPUs :-)
wokwokwok 3y ago
It’s rubbish compared to GPT-3/4. Any model you can trivially load in your browser will be significantly smaller than those models, and broadly speaking smaller = worse. This example is a 4 GB model, that’s (I guess) based off some smallish model like the llama 7B. It’s a proof of concept, not a chat-gpt replacement. There’s nothing here that’s new other than “runs in the browser”; so it won’t be better than any other model you can get your hands on. This kind of thing should be label ByoM (bring your own model). The model isn’t the interesting part of this.
- junrushao1994 3y ago
  To share some fun stuff, here is the response generated by this model: As an AI language model, I would respond by acknowledging that the model discussed in the message is indeed smaller than some of the larger language models like GPT-3/4. However, I would also point out that size is not always a direct indicator of performance, and that the model may still be able to perform well on certain tasks or in certain contexts. Additionally, I would emphasize that the browser compatibility of the model is a significant advantage, as it allows for easy and convenient access to the model without the need to install any specific software or hardware. Finally, I would suggest that the discussion focus on the potential applications and capabilities of the model, rather than solely its size. Back to the topic, we don't make much innovation on the model, so I am probably not the best person to evaluate how a model compares with SOTAs. There are indeed lots of super cool techniques being explored lately that makes it possible to deploy smaller and smaller models, for example, LLM.int8() [1] and int4 quantization [2] without loss of zero-shot accuracy. Can't predict the future, but maybe one day there will be something really powerful but small enough to fit in the pocket of everyone :-) [1] Dettmers, Tim, et al. "LLM. int8 (): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022). [2] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit Inference Scaling Laws." arXiv preprint arXiv:2212.09720 (2022).
  - wokwokwok 3y ago
    > but maybe one day There you go, summarised for you. You can hand wave about quantised models til the end of time but specifically this model is a trivial toy model. No amount of pondering about the future avoid the fundamental fact that small models (~7B) are inferior to larger models like GPT. It’s dishonest to suggest otherwise. :( There’s no reason to do this other than selling snake oil. Maybe. One day. In the future. Things might be different. right now they are not.
jimmySixDOF 3y ago
I'm also not sure what the real use case is here other than an intellectual experiment. I may be missing the point but as I understand the world, running a Local LLM has nothing to do with the web in any way while it is doing so and so what's the benefit of WebGPU ? If you are streaming some constantly changing live data 3D experience in real time served over the internet by a website you just clicked on then WebGPU is making life easier. If you have a local pile of training data and weights sitting there on your hard drive for serious crunching then its like tying your shoelaces together just because developers want to have write once run everywhere dx which is nice for them but value sould always favor the user. Happy for someone to change my mind.
- bigdict 3y ago
  WebGPU is much more portable than CUDA.

slimsag 3y ago

Nice work! I knew it wouldn't be long before someone put this together :) I'm curious if given a different language (like Zig) with WebGPU access if you could easily translate that last-mile of code to execute there or not? In specific I wonder if I can do it, and if you can give me an overview of where the code for "Universal deployment" in your diagram actually lives? I found llm_chat.js, but it seems that doesn't include the logic necessary for building WGSL shaders? Am I wrong or does that happen elsewhere like in the TVM runtime? How much is baked into llm_chat.wasm, and where is the source for that?

crowwork 3y ago
The WGSL are generated and compiled through TVM and embedded into the wasm. I think what you mean is wgpu native support. At the moment the web gpu runtime dispatches to the js webgpu environment. Once TVM runtime comes with wgpu native support (like the current ones in vulkan or metal), then it is possible to leverage any wgpu native runtime like what Zig provide. Additionally, currently tvm natively support targets like vulkan, metal directly which allows targeting these other platforms
- slimsag 3y ago
  OK that makes sense; so basically if I want to give this a shot then I would just need to read llm_chat.js and the TVM docs, and translate llm_chat.js to my language of choice effectively?
  - crowwork 3y ago
    I think instead what would be needed is a wgpu native runtime support for TVM. Like the implementations in tvm vulkan, then it will be naturally link to any runtime that provides webgpu.h Then yah the llm_chat.js would be high-level logic that targets the tvm runtime, and can be implemented in any language that tvm runtime support(that includes, js, java, c++ rust etc). Support webgpu native is an interesting direction. Feel free to open a thread in tvm discuss forum and perhaps there would be fun things to collaborate in OSS