7 ms·
PyTorch for WebGPU
- activatedgeek 3y agoAmazing! Oddly, two tests fail for me with Brave (Version 1.51.118 / Chromium: 113.0.5672.126 (arm64)) on macOS Ventura 13.3.1 - pow([0], [0]) gradient, with "Expected «-Infinity» to be close to «0» (diff: < 0.0000005)" - xlogy([0], [0.30000001192092896]) gradient with "Expected «0» to be close to «-1.2039728164672852»"
- paulddraper 3y agoSame with Chrome 113.0.5672.92 (arm64) on Ventura 13.2. Safari 16.3 has 4 failures: "webgpu is supported", "tensor is webgpu", "xlogy([0], [0]) gradient", "xlogy([0], [0.30000001192092896]) gradient"
- praeclarum 3y agoSorry Safari does not support WebGPU yet. Please join me in writing to Apple and requesting it.
- MuffinFlavored 3y agohttps://praeclarum.org/webgpu-torch/tests/ https://praeclarum.org/webgpu-torch/tests/ This is a dumb question but... are GPUs really that much faster than CPUs specifically at the math functions tested on this page? xlogy trunc tan/tanh sub square sqrt sin/sinc/silu/sinh sign sigmoid sqrt/rsqrt round relu reciprocal rad2deg pow positive neg mul logaddexp/logaddexp2 log/log1p/log10/log2 ldexp hypot frac floor expm1 exp2 exp div deg2rad cos/cosh copysign ceil atan/atan2 asinh/asin add acosh/acos abs Those are the types of math GPUs are good at? I thought they were better at a different kind of math, like matrices or something?
- hedgehog 3y agoThey are relatively tiny but they run on the GPU to avoid lots of copies back and forth.
- nicoco 3y agoGPUs are usually not faster at doing the operation, but excel at doing the operation in parallel on a gazillion elements. Matrix math is mostly additions and multiplications.
- praeclarum 3y agoYeah this is the trick. You need to maximize the use of workgroup parallelism and also lay things out in memory for those kernels to access efficiently. It’s a bit of a balancing act and I’ll be working on benchmarks to test out different strategies.
- hutzlibu 3y agoThe main advantage is parallelism, but on top of that, common math operations are hardware accelerated on the GPU, so should run indeed faster just by being run on the GPU.
- modeless 3y agoGPUs are about 100 times faster than CPUs for any type of single-precision floating point math operation. The catch is that you have to do roughly similar math operations on 10k+ items in parallel before the parallelism and memory bandwidth advantages of the GPU outweigh the latency and single-threaded performance advantages of the CPU. Of course this is achievable in graphics applications with millions of triangles and millions of pixels, and in machine learning applications with millions or billions of neurons. IMO almost any application that is bottlenecked by CPU performance can be recast to use GPUs effectively. But it's rarely done because GPUs aren't nearly as standardized as CPUs and the developer tools are much worse, so it's a lot of effort for a faster but much less portable outcome.
- HexDecOctBin 3y agoAre there any standardised approaches for this? I fail to imagine how one would put branchy CPU code like parsing, etc. on GPUs effectively?
- raphlinus 3y agoIt is possible but you have to do things very differently, for example use monoids. There are a few compilers implemented on GPU, including Aaron Hsu's co-dfns and Voetter's compiler project[1]. The parentheses matching problem itself (the core of parsing) has long known efficient parallel algorithms and those have been ported to compute shaders[2] (disclosure: blatant self-promotion). [1]: https://dl.acm.org/doi/pdf/10.1145/3528416.3530249 https://dl.acm.org/doi/pdf/10.1145/3528416.3530249 [2]: https://arxiv.org/pdf/2205.11659.pdf https://arxiv.org/pdf/2205.11659.pdf
- kaliqt 3y agoWebGPU I think will help change a lot of this. Finally, portable code that is performant and runs virtually anywhere. It's the same reason web apps have taken off so much, or just the idea of deploying to and from web platforms, e.g. write in web and deploy to native. I think WebGPU will be that universal language everyone speaks, and I think also that this will help get rid of Nvidia's monopoly on GPU compute.
- 3y ago
- praeclarum 3y agoYeah so the thing is WebGPU doesn’t correctly support IEEE floating point. Particularly, 0 is often substituted for +-Inf and NaN. See section 14.6 of the spec. https://www.w3.org/TR/WGSL/#floating-point-evaluation https://www.w3.org/TR/WGSL/#floating-point-evaluation It’s not such a problem for real nets since you avoid those values like the plague. But the tests catch them and I need to make the tests are tolerant. Thanks for the results!
- replete 3y agoImpressive work. A number of test failures for me on chromium 113.0.5672.63 (ungoogled chromium) MacOS Ventura 13.3.1: https://pastebin.com/eM6ZA3j2 https://pastebin.com/eM6ZA3j2 I'll open a ticket if it helps..
- praeclarum 3y agoPlease do. I have a few test machines but cannot match the variety of hardware out there.
- sebzim4500 3y agoLoads of tests fail for me (chrome, windows). Mainly trigonometric functions which are way less accurate than they are supposed to be.
- praeclarum 3y agoYeah I think I’ll reduce the accuracy requirement for some transcendental functions since GPUs seem all over the place.
- garbagecoder 3y agoThere goes my weekend!! Thanks!
- quickthrower2 3y agoLearning as a beginner/novice, Feels like I am trying to catch up to a jet at takeoff speed on my kick scooter.
- sva_ 3y agoVery impressive work. Would be interesting to do some benchmarks versus PyTorch. On a side-note, I'm not sure if it is because I've looked at so many autograd engines by now, but it is really cool to see that after the years of different frameworks having been developed, most people seem to agree on some concepts and structure on how to implement something like this. It is pretty easy to dive into this, even without being particularly skilled in JS/TS. Wondering how such frameworks will look in a couple years.
- westurner 3y agoCould there be something like emscripten-forge/requests-wasm-polyfill for PyTorch with WebGPU? https://github.com/emscripten-forge/requests-wasm-polyfill https://github.com/emscripten-forge/requests-wasm-polyfill How does the performance of webgpu-torch compare to compiling PyTorch to WASM with emscripten and WebGPU? tfjs benchmarks: Environment > backend > {WASM, WebGL, CPU, WebGPU, tflite} https://tensorflow.github.io/tfjs/e2e/benchmarks/local-benchmark/index.html https://tensorflow.github.io/tfjs/e2e/benchmarks/local-bench... src: https://github.com/tensorflow/tfjs/tree/master/e2e/benchmarks https://github.com/tensorflow/tfjs/tree/master/e2e/benchmark... tensorflow/tfjs https://github.com/tensorflow/tfjs https://github.com/tensorflow/tfjs tfjs-backend-wasm https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-wasm https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-... tfjs-backend-webgpu https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-webgpu https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-... ([...], tflite-support, tflite-micro) From facebookresearch/shumai (a JS tensor library) https://github.com/facebookresearch/shumai/issues/122 https://github.com/facebookresearch/shumai/issues/122 : > It doesn't make sense to support anything besides WebGPU at this point. WASM + SIMD is around 15-20x slower on my machine[1]. Although WebGL is more widely supported today, it doesn't have the compute features needed for efficient modern ML (transformers etc) and will likely be a deprecated backend for other frameworks when WebGPU comes online. tensorflow rust has a struct.Tensor: https://tensorflow.github.io/rust/tensorflow/struct.Tensor.html https://tensorflow.github.io/rust/tensorflow/struct.Tensor.h... "ONNX Runtime merges WebGPU backend" https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime https://news.ycombinator.com/item?id=35696031 https://news.ycombinator.com/item?id=35696031 ... TIL about wonnx: https://github.com/webonnx/wonnx#in-the-browser-using-webgpu--webassembly https://github.com/webonnx/wonnx#in-the-browser-using-webgpu... microsoft/onnxruntime: https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime Apache/arrow has language-portable Tensors for cpp: https://arrow.apache.org/docs/cpp/api/tensor.html https://arrow.apache.org/docs/cpp/api/tensor.html and rust: https://docs.rs/arrow/latest/arrow/tensor/struct.Tensor.html https://docs.rs/arrow/latest/arrow/tensor/struct.Tensor.html and Python: https://arrow.apache.org/docs/python/api/tables.html#tensors https://arrow.apache.org/docs/python/api/tables.html#tensors https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor https://arrow.apache.org/docs/python/generated/pyarrow.Tenso... Fwiw it looks like the llama.cpp Tensor is from ggml, for which there are CUDA and OpenCL implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h https://github.com/ggerganov/llama.cpp/blob/master/ggml.h Are the recommendable ways to cast e.g. arrow Tensors to pytorch/tensorflow? FWIU, Rust has a better compilation to WASM; and that's probably faster than already-compiled-to-JS/ES TensorFlow + WebGPU. What's a fair benchmark?
- uoaei 3y ago> This is a perfect scenario to take advantage of code generation. I wrote a code generator that takes a template and generates the optimized kernels for each operation. The code generator is written in TypeScript and generates WebGPU compute shader code. This means that the generated code can be heavily optimized for the given scenario and those optimizations can be shared between operations. A clever way to implement an AOT variant of the operator fusion methods in the XLA (JIT) compiler.
- SloopJon 3y agoThis is perhaps the most interesting aspect of the project--using a code generator to escape the gravitational pull of CUDA. I wonder how well it would generalize to other targets.
- crubier 3y agoThis is huge! For me the one thing preventing Typescript to replace python is the lack of availability of CV ML libraries. WebGPU and this kind of libraries changes everything
- tzhenghao 3y agoThis. Just to riff off an example, a lot of the APIs in common DL frameworks like PyTorch revolve around numpy or pickle formats. These are Python first semantics.
- lyu07282 3y agoThere is so much stuff in scipy and opencv alone that it will take forever for another language to catch up to. Unfortunately, because python is suuuuuuuch a mediocre language in comparison. Type annotations were such a lost opportunity in python, it's such a horrible implementation.
- SuboptimalEng 3y agoAnd operator overloading. TS code tends to look like this `c.add(b.add(a))` or `add(add(a, b), c)` instead of `a + b + c` as you might write in Python. That was my biggest pain-point with using TS for graphics related projects. If operator overloading existed, then TS would be a no brainer for entry level graphics + AI/ML projects. Edit: This gets more complicated when doing operations that force you to manually respect PEMDAS. For example, `add(div(a, b), multiply(c, d))` in TypeScript would simplify to `a / b + c * d` in Python. The TS version is unreadable.
- killthebuddha 3y agoAnother option that's not quite as good as `a + b + c` but that is possible with TypeScript is a fluent API: const sum = a.add(b).add(c);
- leeoniya 3y agoor add(a,b,c)
- newhouseb 3y agoI'm excited about this for probably different reasons than most: I think Typescript could be a more ergonomic way to develop ML models than Python because you can automatically infer and check tensor dimensions while you are writing code! Compare this to the mess of comments you usually see writing pytorch telling you that x is of shape [x, y, z]. // An empty 3x4 matrix const tensorA = tensor([3, 4]) // An empty 4x5 matrix const tensorB = tensor([4, 5]) const good = multiplyMatrix(tensorA, tensorB); ^ Inferred type is Tensor<readonly [3, 5]> const bad = multiplyMatrix(tensorB, tensorA); ^^^^^^^ Argument of type 'Tensor<readonly [4, 5]>' is not assignable to parameter of type '[never, "Differing types", 3 | 5]'.(2345) I prototyped this for PotatoGPT [1] and some kind stranger on the internet wrote up a more extensive take [2]. You can play with an early version on the Typescript playground here [3] (uses a twitter shortlink for brevity) [1] https://github.com/newhouseb/potatogpt https://github.com/newhouseb/potatogpt [2] https://sebinsua.com/type-safe-tensors https://sebinsua.com/type-safe-tensors [3] https://t.co/gUzzTl4AAN https://t.co/gUzzTl4AAN
- whimsicalism 3y agoThat work looks really interesting! I am also excited about type safety when it comes to tensors. My understanding was that this type safe approach to tensor shape had encountered issues because it was difficult/impossible (maybe?) to reason about the shape of some common operators at compile time. But perhaps those operators are not really necessary. [0] Some sort of typed 'named tensor' that could be combined with einsum notation at runtime would be awesome, ie. (don't really know TS/JS well but pseudocode) import { torch } from 'pytorch' as t import { torch.nn } from 'pytorch' as nn const tensorA: Tensor[Batch, Seq, Emb] = t.randn([10,10,10]) // initialize tensor const transformLayer = nn.Einsum((Batch, Seq, Emb),(Emb)->(Batch, Seq)) const tensorB: Tensor[Emb2] = t.randn([20]) const transformedOutput = transformLayer(tensorA, tensorB) // type error: Emb2 does not match Emb [0]: https://github.com/pytorch/pytorch/issues/26889 https://github.com/pytorch/pytorch/issues/26889
- newhouseb 3y ago
- foota 3y agoIt seems like there's a developing competitor to the Python ecosystem in the form of webgpu and js/ts. Being able to run anywhere with no native dependencies is a pretty huge advantage, it will be interesting to see if this steals momentum. I wonder how hard it would be to add support for this as an alternate backend to transformers.js.
- hot_gril 3y agoI used to use Python heavily and favor JS now for these reasons. Portability is huge. I think JS is going to eat Python's lunch.
- quickthrower2 3y agoImagine: Isomorphic neural nets that can run server or client side.
- nwoli 3y agoI would love to hear what Bram Wasti thinks about this (who has experience in this area and sometimes frequents the HN comments).
- tachim 3y agoWhat's the reason to run pytorch directly on WebGPU vs using ONNX on WebGPU (e.g. with https://github.com/webonnx/wonnx https://github.com/webonnx/wonnx)?
- infrawhispers 3y agoThis is really nice! I have been working on getting ANN search working in the browser ([1] demo, [2] WIP repo) and would love to switch out onnx for the embedding generation. [1] https://anansi.pages.dev/ https://anansi.pages.dev/ [2] https://github.com/infrawhispers/anansi/tree/main/embedds/lib/retrieval/src https://github.com/infrawhispers/anansi/tree/main/embedds/li... privacy focused semantic search / ML at the edge is looking brighter every day.
- vrglvrglvrgl 3y ago[dead]
- zmmmmm 3y agoCurious what the potential is for this to then run headless - is the support for this in chrome etc. built into v8 etc such that node and others can simply piggyback on it? Or is it sitting in the browser layer such that you'd have to end up with a headless browser or similar?
- henearkr 3y agoNode doesn't have a GPU backend. Deno has (or had), but you'd have to use Deno v1.31.3 to get WebGPU support (because if was removed afterwards for startup performance issues).