4 ms·

There's also the huge caveat of CUDA project compatibility. AMD/Intel seem to be a more promising in that regard, and thats not saying much. Not that it isn't

There's also the huge caveat of CUDA project compatibility. AMD/Intel seem to be a more promising in that regard, and thats not saying much. Not that it isn't appealing... If the price is right. Also, both AMD and Intel are both reportedly coming out with M Pro-like SoCs. AMD's "Strix Halo" is rumored for 2025. Less is known about Intel's Arrow Lake SKU at the moment.

sangnoir 3y ago
How many people are writing their own CUDA kernels? I think Pytorch compatibility is a bigger win for AMD amd Intel, when solving for the vast majority of AI practitioners who live in Jupyter Notebooks
- brucethemoose2 3y ago
  I don't see the distinction. Tons of popular PyTorch projects run hand made CUDA kernels, whether through a library or one written themselves. In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch, but the performance hit is often severe, and not many inference projects are using torch.compile or triton kernels to compensate.
  - sangnoir 3y ago
    > I don't see the distinction Performant Pytorch compatibility can be achieved without CUDA compatibility. > In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch This is the crux of my point: AMD ought to focus on fixing these Pytorch incompatibilities instead of splitting focus with chasing CUDA translations.
- Baldbvrhunter 3y ago
  Greater than zero. Julia has a CUDA compiling and I've hand crafted CUDA kernels myself. https://cuda.juliagpu.org/stable/ https://cuda.juliagpu.org/stable/
  - ChrisRackauckas 3y ago
    Indeed, and this year we created a system for compiling ODE code not just optimized CUDA kernels but also OneAPI kernels, AMD GPU kernels, and Metal. Peer reviewed version is here (https://www.sciencedirect.com/science/article/abs/pii/S0045782523007156 https://www.sciencedirect.com/science/article/abs/pii/S00457...), open access is here (https://arxiv.org/abs/2304.06835 https://arxiv.org/abs/2304.06835), and the open source code is at https://github.com/SciML/DiffEqGPU.jl https://github.com/SciML/DiffEqGPU.jl. The key that the paper describes is that in this case kernel generation is about 20x-100x faster than PyTorch and Jax (see the Jax compilation in multiple ways in this notebook https://colab.research.google.com/drive/1d7G-O5JX31lHbg7jTzzozbo5-Gp7DBEv?usp=sharing https://colab.research.google.com/drive/1d7G-O5JX31lHbg7jTzz..., extra overhead though from calling Julia from Python but still shows a 10x). The point really is that while deep learning libraries are amazing, at the end of the day they are DSL and really pull towards one specific way of computing and parallelization. It turns out that way of parallelizing is good for deep learning, but not for all things you may want to accelerate. Sometimes (i.e. cases that aren't dominated by large linear algebra) building problem-specific kernels is a major win, and it's over-extrapolating to see ML frameworks do well with GPUs and think that's the only thing that's required. There are many ways to parallelize a code, ML libraries hardcode a very specific way, and it's good for what they are used for but not every problem that can arise.

sangnoir 3y ago

How many people are writing their own CUDA kernels? I think Pytorch compatibility is a bigger win for AMD amd Intel, when solving for the vast majority of AI practitioners who live in Jupyter Notebooks

brucethemoose2 3y ago
I don't see the distinction. Tons of popular PyTorch projects run hand made CUDA kernels, whether through a library or one written themselves. In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch, but the performance hit is often severe, and not many inference projects are using torch.compile or triton kernels to compensate.
- sangnoir 3y ago
  > I don't see the distinction Performant Pytorch compatibility can be achieved without CUDA compatibility. > In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch This is the crux of my point: AMD ought to focus on fixing these Pytorch incompatibilities instead of splitting focus with chasing CUDA translations.
Baldbvrhunter 3y ago
Greater than zero. Julia has a CUDA compiling and I've hand crafted CUDA kernels myself. https://cuda.juliagpu.org/stable/ https://cuda.juliagpu.org/stable/
- ChrisRackauckas 3y ago
  Indeed, and this year we created a system for compiling ODE code not just optimized CUDA kernels but also OneAPI kernels, AMD GPU kernels, and Metal. Peer reviewed version is here (https://www.sciencedirect.com/science/article/abs/pii/S0045782523007156 https://www.sciencedirect.com/science/article/abs/pii/S00457...), open access is here (https://arxiv.org/abs/2304.06835 https://arxiv.org/abs/2304.06835), and the open source code is at https://github.com/SciML/DiffEqGPU.jl https://github.com/SciML/DiffEqGPU.jl. The key that the paper describes is that in this case kernel generation is about 20x-100x faster than PyTorch and Jax (see the Jax compilation in multiple ways in this notebook https://colab.research.google.com/drive/1d7G-O5JX31lHbg7jTzzozbo5-Gp7DBEv?usp=sharing https://colab.research.google.com/drive/1d7G-O5JX31lHbg7jTzz..., extra overhead though from calling Julia from Python but still shows a 10x). The point really is that while deep learning libraries are amazing, at the end of the day they are DSL and really pull towards one specific way of computing and parallelization. It turns out that way of parallelizing is good for deep learning, but not for all things you may want to accelerate. Sometimes (i.e. cases that aren't dominated by large linear algebra) building problem-specific kernels is a major win, and it's over-extrapolating to see ML frameworks do well with GPUs and think that's the only thing that's required. There are many ways to parallelize a code, ML libraries hardcode a very specific way, and it's good for what they are used for but not every problem that can arise.