6 ms·

This is very exciting! (I had suspected it would slip to 114) WebGPU implementations are still pretty immature, but certainly enough to get started with. I've

This is very exciting! (I had suspected it would slip to 114) WebGPU implementations are still pretty immature, but certainly enough to get started with. I've been implementing a Rust + WebGPU ML runtime for the past few months and have enjoyed writing WGSL. I recently got a 250M parameter LLM running in the browser without much optimisation and it performs pretty well! (https://twitter.com/fleetwood___/status/1638469392794091520 https://twitter.com/fleetwood___/status/1638469392794091520) That said, matmuls are still pretty handicapped in the browser (especially considering the bounds checking enforced in the browser). From my benchmarking I've struggled to hit 50% of theoretical FLOPS, which is cut down to 30% when the bounds checking comes in. (Benchmarks here: https://github.com/FL33TW00D/wgpu-mm https://github.com/FL33TW00D/wgpu-mm) I look forward to accessing shader cores as they mentioned in the post.

tehsauce 4y ago
It’s better to compare against an ML framework than to maximum theoretical flops because sometimes it’s not possible to reach. These models are often limited by memory bandwidth rather than flop capability.
- pklausler 4y ago
  Something more than 30 years ago, I had the privilege of working as a young compiler writer for [a supercomputer designer]'s penultimate start-up. I once naively asked him why the machine he was working on couldn't have more memory bandwidth, since the floating-point functional units were sometimes starved for operand data and it was hard to hit the peak Flop/sec figures. And his response has stuck with me ever since; basically, it makes better sense for the memory to be fully utilized, not the floating-point units, because the memory paths were way more expensive than the floating-point units. And this was something you could actually physically see through the transparent top of the system's case. I guess the lesson would be: Don't let a constraint that would be fairly cheap to overdesign be the limiting factor in a system's performance.
singularity2001 4y ago
How would WebGL matmul fare in comparison?
- FL33TW00D 4y ago
  This blog post breaks it down pretty well: https://pixelscommander.com/javascript/webgpu-computations-performance-in-comparison-to-webgl/ https://pixelscommander.com/javascript/webgpu-computations-p...
misterdata 4y ago
Looking forward to your WebGPU ML runtime! Also, why not contribute back to WONNX? (https://github.com/webonnx/wonnx https://github.com/webonnx/wonnx)
- FL33TW00D 4y ago
  Hi Tommy! I sent you an email a few weeks back - would be great to chat! WONNX is a seriously impressive project. There is a few reason I didn't just contribute back to WONNX: 1. WONNX does not parse the ONNX model into an IR, which I think is essential to have the freedom to transform the model as required. 2. When I started, WONNX didn't seem focused on symbolic dimensions (but I've seen you shipping the shape inference recently!). 3. The code quality has to be much higher when it's open source! I wanted to hack on this without anyone to please but myself.
  - antimora 4y ago
    I'm presently working on enhancing Burn's (https://burn-rs.github.io/ https://burn-rs.github.io/) capabilities by implementing ONNX model importation (https://github.com/burn-rs/burn/issues/204 https://github.com/burn-rs/burn/issues/204). This will enable users to generate model source code during build time and load weights at runtime. In my opinion, ONNX is more complex than necessary. Therefore, I opted to convert it to an intermediate representation (IR) first, which is then used to generate source code. A key advantage of this approach is the ease of merging nodes into corresponding operations, since ONNX and Burn don't share the same set of operators.
    - FL33TW00D 4y ago
      Looks great! ONNX is 100% more complex than necessary. Another format of interest is NNEF: https://www.khronos.org/nnef https://www.khronos.org/nnef
      - misterdata 4y ago
        Also see the recently introduced StableHLO and its serialization format: https://github.com/openxla/stablehlo/blob/main/docs/bytecode.md https://github.com/openxla/stablehlo/blob/main/docs/bytecode...
MuffinFlavored 4y ago
what would it take to python -> wasm -> webgpu for the entire existing webgpu ecosystem (all of the libraries around neural networks, torch, yada yada)
- grlass 4y ago
  The Apache TVM machine learning compiler has a WASM and WebGPU backend, and can import from most DNN frameworks. Here's a project running Stable Diffusion with webgpu and TVM [1]. Questions exist around post-and-pre-processing code in folks' Python stacks, with e.g. NumPy and opencv. There's some NumPy to JS transpilers out there, but those aren't feature complete or fully integrated. [1] https://github.com/mlc-ai/web-stable-diffusion https://github.com/mlc-ai/web-stable-diffusion
- korijn 4y ago
  FYI you can already use webgpu directly in python, see https://github.com/pygfx/wgpu-py https://github.com/pygfx/wgpu-py for webgpu wrappers and https://github.com/pygfx/pygfx https://github.com/pygfx/pygfx for a more high level graphics library
  - MuffinFlavored 4y ago
    I think I meant to ask "how is machine learning support for things like neural networks instead of graphical 2D/3D operations with WebGPU"?
    - korijn 4y ago
      Well, you could implement that on top of the wrappers as well I guess. Anyway!
      - MuffinFlavored 4y ago
        As in, I could, but a library/libraries doing so do not yet exist?
brrrrrm 4y ago
> WebGPU ML runtime oh cool! will this be numpy-like or will it have autograd as well? We're looking around for a web backend for shumai[1] and the former is really all we need :) [1]: https://github.com/facebookresearch/shumai https://github.com/facebookresearch/shumai
- FL33TW00D 4y ago
  Autograd would be a whole new adventure, just inference for now :).
antimora 4y ago
Oh great! I am one of the contributors for Burn (Rust Deep Learning Framework). We have a plan adding a WebGPU backend (https://github.com/burn-rs/burn/issues/243 https://github.com/burn-rs/burn/issues/243). Here is more about the framework Burn: https://burn-rs.github.io/ https://burn-rs.github.io/
- lacker 4y ago
  Burn looks interesting... so I notice in the sample code you mention: use burn_tch::TchBackend; I don't see "TchBackend" mentioned in the docs. Is Tch = torch and Burn can be used as a Torch (or tch-rs) wrapper? Or am I imagining too much from a miscellaneous three letter prefix? I've been looking for a good way to do standard modern AI stuff in Rust, and was leaning towards tch-rs because it exists and seems reasonably frequently maintained, but if you have some sort of reason to use Burn instead then I'm curious to hear it.
  - antimora 4y ago
    That's right, Tch stands for torch. We use tch-rs (bindings to torch++). The selling point of burn is that it offers the full level of deep learning framework with ability to swap backends. This is useful, for example, I can train with Torch backend on GPUs but if I want to deploy, lets say to embedded device, I can use NDArray backend, which supports pure rust code (also with various BLAS acceleration if needed). You can also use NDArray backend with Accelerate (iOS or MacOS blass). You can even compile for WASM because Burn supports inference with no_std (you can try this online demo https://burn-rs.github.io/demo https://burn-rs.github.io/demo). The framework is well written and follows Rust's conventions and best practices. It's still evolving, however. Give it a try. If you find problems, you can file a ticket or join discord chat.
jimmySixDOF 4y ago
I guess all the AI type use cases are front seat here but the performance boost increase to Immersive Web (WebXR) and general moves towards webpages as 3D UI/UX for applications -- that is where I hope to see expansion into new creative territory.
winter_blue 4y ago
Does the Rust compile to WebAssembly? I’m guessing that means WebGPU is fully accessible from WebAssembly, and one can go a zero JS route?
shaklee3 4y ago
for a lot of gemm sizes 50% is quite good. you'll only hit peak with the exact optimal sizes and/or batches

tehsauce 4y ago

It’s better to compare against an ML framework than to maximum theoretical flops because sometimes it’s not possible to reach. These models are often limited by memory bandwidth rather than flop capability.

pklausler 4y ago
Something more than 30 years ago, I had the privilege of working as a young compiler writer for [a supercomputer designer]'s penultimate start-up. I once naively asked him why the machine he was working on couldn't have more memory bandwidth, since the floating-point functional units were sometimes starved for operand data and it was hard to hit the peak Flop/sec figures. And his response has stuck with me ever since; basically, it makes better sense for the memory to be fully utilized, not the floating-point units, because the memory paths were way more expensive than the floating-point units. And this was something you could actually physically see through the transparent top of the system's case. I guess the lesson would be: Don't let a constraint that would be fairly cheap to overdesign be the limiting factor in a system's performance.

singularity2001 4y ago

How would WebGL matmul fare in comparison?

FL33TW00D 4y ago
This blog post breaks it down pretty well: https://pixelscommander.com/javascript/webgpu-computations-performance-in-comparison-to-webgl/ https://pixelscommander.com/javascript/webgpu-computations-p...

misterdata 4y ago

Looking forward to your WebGPU ML runtime! Also, why not contribute back to WONNX? (https://github.com/webonnx/wonnx https://github.com/webonnx/wonnx)

FL33TW00D 4y ago
Hi Tommy! I sent you an email a few weeks back - would be great to chat! WONNX is a seriously impressive project. There is a few reason I didn't just contribute back to WONNX: 1. WONNX does not parse the ONNX model into an IR, which I think is essential to have the freedom to transform the model as required. 2. When I started, WONNX didn't seem focused on symbolic dimensions (but I've seen you shipping the shape inference recently!). 3. The code quality has to be much higher when it's open source! I wanted to hack on this without anyone to please but myself.
- antimora 4y ago
  I'm presently working on enhancing Burn's (https://burn-rs.github.io/ https://burn-rs.github.io/) capabilities by implementing ONNX model importation (https://github.com/burn-rs/burn/issues/204 https://github.com/burn-rs/burn/issues/204). This will enable users to generate model source code during build time and load weights at runtime. In my opinion, ONNX is more complex than necessary. Therefore, I opted to convert it to an intermediate representation (IR) first, which is then used to generate source code. A key advantage of this approach is the ease of merging nodes into corresponding operations, since ONNX and Burn don't share the same set of operators.
  - FL33TW00D 4y ago
    Looks great! ONNX is 100% more complex than necessary. Another format of interest is NNEF: https://www.khronos.org/nnef https://www.khronos.org/nnef
    - misterdata 4y ago
      Also see the recently introduced StableHLO and its serialization format: https://github.com/openxla/stablehlo/blob/main/docs/bytecode.md https://github.com/openxla/stablehlo/blob/main/docs/bytecode...

MuffinFlavored 4y ago

what would it take to python -> wasm -> webgpu for the entire existing webgpu ecosystem (all of the libraries around neural networks, torch, yada yada)

grlass 4y ago
The Apache TVM machine learning compiler has a WASM and WebGPU backend, and can import from most DNN frameworks. Here's a project running Stable Diffusion with webgpu and TVM [1]. Questions exist around post-and-pre-processing code in folks' Python stacks, with e.g. NumPy and opencv. There's some NumPy to JS transpilers out there, but those aren't feature complete or fully integrated. [1] https://github.com/mlc-ai/web-stable-diffusion https://github.com/mlc-ai/web-stable-diffusion
korijn 4y ago
FYI you can already use webgpu directly in python, see https://github.com/pygfx/wgpu-py https://github.com/pygfx/wgpu-py for webgpu wrappers and https://github.com/pygfx/pygfx https://github.com/pygfx/pygfx for a more high level graphics library
- MuffinFlavored 4y ago
  I think I meant to ask "how is machine learning support for things like neural networks instead of graphical 2D/3D operations with WebGPU"?
  - korijn 4y ago
    Well, you could implement that on top of the wrappers as well I guess. Anyway!
    - MuffinFlavored 4y ago
      As in, I could, but a library/libraries doing so do not yet exist?

brrrrrm 4y ago

> WebGPU ML runtime oh cool! will this be numpy-like or will it have autograd as well? We're looking around for a web backend for shumai[1] and the former is really all we need :) [1]: https://github.com/facebookresearch/shumai https://github.com/facebookresearch/shumai

FL33TW00D 4y ago
Autograd would be a whole new adventure, just inference for now :).

antimora 4y ago

Oh great! I am one of the contributors for Burn (Rust Deep Learning Framework). We have a plan adding a WebGPU backend (https://github.com/burn-rs/burn/issues/243 https://github.com/burn-rs/burn/issues/243). Here is more about the framework Burn: https://burn-rs.github.io/ https://burn-rs.github.io/

lacker 4y ago
Burn looks interesting... so I notice in the sample code you mention: use burn_tch::TchBackend; I don't see "TchBackend" mentioned in the docs. Is Tch = torch and Burn can be used as a Torch (or tch-rs) wrapper? Or am I imagining too much from a miscellaneous three letter prefix? I've been looking for a good way to do standard modern AI stuff in Rust, and was leaning towards tch-rs because it exists and seems reasonably frequently maintained, but if you have some sort of reason to use Burn instead then I'm curious to hear it.
- antimora 4y ago
  That's right, Tch stands for torch. We use tch-rs (bindings to torch++). The selling point of burn is that it offers the full level of deep learning framework with ability to swap backends. This is useful, for example, I can train with Torch backend on GPUs but if I want to deploy, lets say to embedded device, I can use NDArray backend, which supports pure rust code (also with various BLAS acceleration if needed). You can also use NDArray backend with Accelerate (iOS or MacOS blass). You can even compile for WASM because Burn supports inference with no_std (you can try this online demo https://burn-rs.github.io/demo https://burn-rs.github.io/demo). The framework is well written and follows Rust's conventions and best practices. It's still evolving, however. Give it a try. If you find problems, you can file a ticket or join discord chat.

jimmySixDOF 4y ago

I guess all the AI type use cases are front seat here but the performance boost increase to Immersive Web (WebXR) and general moves towards webpages as 3D UI/UX for applications -- that is where I hope to see expansion into new creative territory.

winter_blue 4y ago

Does the Rust compile to WebAssembly? I’m guessing that means WebGPU is fully accessible from WebAssembly, and one can go a zero JS route?

shaklee3 4y ago

for a lot of gemm sizes 50% is quite good. you'll only hit peak with the exact optimal sizes and/or batches