4 ms·

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Meta

I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library (or find said higher level library all together). Have you had any luck? In my experience, it's basically Python or bust in this space despite lots of efforts to make it not that way? I also got confuses as to whether a 'shader' was more for the visual GPU output of things, or if it was also a building block for model training/networks/machine learning/etc.

smoldesu 4y ago
Give this a look: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert/blob/master/examples/generation_gpt_neo.rs https://github.com/guillaume-be/rust-bert/blob/master/exampl... If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it. > it's basically Python or bust in this space More or less, but that doesn't have to be a bad thing. If you're on Apple Silicon, you have plenty of performance headroom to deploy Python code for this. I've gotten this library to work on systems with as little as 2gb of memory, so outside of ultra-low-end use cases, you should be fine.
- MuffinFlavored 4y ago
  To clarify, > Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing from rust-tokenizers. > tch-rs: Rust bindings for the C++ api of PyTorch. Which "backend" does this end up using on Apple Silicon, MPS (Metal Performance Shaders) or OpenCL? https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html I'm going to guess MPS?
  - smoldesu 4y ago
    Whatever your Pytorch install is designed to accelerate. I've got Ampere-accelerated Pytorch running it on my ARM server, I assume MPS is used on compatible systems.
fathyb 4y ago
> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less convenient than with Swift/Objective-C, but still possible. Worst case you'll have to add an .mm file and bridge it with `extern "C"`. That said, doing GPGPU is not doing ML, and most ML libraries are in Python. > I also got confuses as to whether a 'shader' was more for the visual GPU output of things, or if it was also a building block for model training/networks/machine learning/etc. A shader is basically a function that runs for every element of the output buffer. We generally call them kernels for GPGPU, and shaders (geometry, vertex, fragment) for graphics stuff. You have to write them in a language that kinda looks like C (OpenGL GLSL, DirectX HSL, Metal MSL), but is optimized for the SMT properties of GPUs. Learning shaders will let you run code on the GPU, to do ML you also need to learn what are tensors, how to compute them on the GPU, and how to build ML systems using them. I recommend ShaderToy [0] if you want a cool way to understand and play with shaders. [0]: https://www.shadertoy.com/ https://www.shadertoy.com/
- MuffinFlavored 4y ago
  so write a kernel in OpenCL, then call it from Rust is that what machine learning is doing at a high level?
  - fathyb 4y ago
    At a very high level yes. There is also the very important step of efficiently laying out data in the GPU memory to compute tensor values in the kernels.
    - MuffinFlavored 4y ago
      Can you confirm if OpenCL has been deprecated going forward for Apple Silicon please? Also, should I be able to expect to use OpenCL version 3.0 on Apple Silicon, or only v1.2 or 2.0 or something else?

smoldesu 4y ago

Give this a look: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert/blob/master/examples/generation_gpt_neo.rs https://github.com/guillaume-be/rust-bert/blob/master/exampl... If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it. > it's basically Python or bust in this space More or less, but that doesn't have to be a bad thing. If you're on Apple Silicon, you have plenty of performance headroom to deploy Python code for this. I've gotten this library to work on systems with as little as 2gb of memory, so outside of ultra-low-end use cases, you should be fine.

MuffinFlavored 4y ago
To clarify, > Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing from rust-tokenizers. > tch-rs: Rust bindings for the C++ api of PyTorch. Which "backend" does this end up using on Apple Silicon, MPS (Metal Performance Shaders) or OpenCL? https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html I'm going to guess MPS?
- smoldesu 4y ago
  Whatever your Pytorch install is designed to accelerate. I've got Ampere-accelerated Pytorch running it on my ARM server, I assume MPS is used on compatible systems.

fathyb 4y ago

> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less convenient than with Swift/Objective-C, but still possible. Worst case you'll have to add an .mm file and bridge it with `extern "C"`. That said, doing GPGPU is not doing ML, and most ML libraries are in Python. > I also got confuses as to whether a 'shader' was more for the visual GPU output of things, or if it was also a building block for model training/networks/machine learning/etc. A shader is basically a function that runs for every element of the output buffer. We generally call them kernels for GPGPU, and shaders (geometry, vertex, fragment) for graphics stuff. You have to write them in a language that kinda looks like C (OpenGL GLSL, DirectX HSL, Metal MSL), but is optimized for the SMT properties of GPUs. Learning shaders will let you run code on the GPU, to do ML you also need to learn what are tensors, how to compute them on the GPU, and how to build ML systems using them. I recommend ShaderToy [0] if you want a cool way to understand and play with shaders. [0]: https://www.shadertoy.com/ https://www.shadertoy.com/

MuffinFlavored 4y ago
so write a kernel in OpenCL, then call it from Rust is that what machine learning is doing at a high level?
- fathyb 4y ago
  At a very high level yes. There is also the very important step of efficiently laying out data in the GPU memory to compute tensor values in the kernels.
  - MuffinFlavored 4y ago
    Can you confirm if OpenCL has been deprecated going forward for Apple Silicon please? Also, should I be able to expect to use OpenCL version 3.0 on Apple Silicon, or only v1.2 or 2.0 or something else?