2 ms·
I don't usually like to do this, because clearly someone wrote this post (or at the very least they cared enough to have a LLM help draft it :P). Maybe it's jus
by saagarjha 2y ago
I don't usually like to do this, because clearly someone wrote this post (or at the very least they cared enough to have a LLM help draft it :P). Maybe it's just me and my interests not being aligned with the content. But I just found myself kind of disappointed by the content?
Like, the post is about how you can do GEMM tuning on AMD GPUs, a subject which is inherently super interesting–there's a lot of nuance to writing optimal kernels, and some of this is expressed in the article, too. Combine that with an architecture that isn't Nvidia? It's an excellent setup for something that would be interesting to read.
Which makes the actual conclusion all the more disappointing, IMO. There's nothing about what the actual optimizations are. It's just "oh yeah we tuned our GEMMs and now LLaMA is faster". Like, I get that nobody actually cares about GEMM and they just want tokens to come out of their GPU. But still, that's like writing a blog post about how you can speed up your game with SIMD and then posting some charts of how Cyberpunk 2077 gives you 2x the frame rate now. Ok, but how? I just feel like the interesting part is missing.
- xoranth 2y agoI believe the reason they offer no details about how they tuned the kernels is that the tuning is done by a tool provided by AMD. See here: https://rocm.docs.amd.com/projects/rocBLAS/en/develop/how-to/Programmers_Guide.html#rocblas-gemm-tune https://rocm.docs.amd.com/projects/rocBLAS/en/develop/how-to...
- latchkey 2y agoThis article builds upon the work that C&C just released an article on. https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/ https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3... Elio, the person who did the testing, worked together with C&C, on my system. He then took it to the next level with this gemm work. The docker container is linked on the article now so that anyone can reproduce the results.