4 ms·

I see they have experimental AMD backends that don’t use ROCm. Is ROCm that bad that they wrote their own, or was there some other justification for this?

by jarbus 2y ago

blihp 2y ago
The author of the library has done numerous YouTube rants on how bad he thinks the AMD compute drivers are.
- JMiao 2y ago
  What does he get wrong?
  - KaoruAoiShiho 2y ago
    He's not wrong.
- roenxi 2y ago
  Has he? The ones I'm aware of he was complaining about the low quality of the generic kernel drivers. AMD software had a tendency to crash when doing anything outside of standard video games (which was my experience too, but I've caved and bought Nvidia since then; average driver quality of Nvidia on linux seems to be much lower but the kernel doesn't go down which is nice. Got a lot of OOM errors where on ROCm the kernel froze requiring a full system restart). But this is interesting and probably strong evidence that the CUDA API isn't the moat people thought it was. CUDA multiplies matricies and that is close to a commodity operation. The moat actually seems to be Nvidia's higher generic software engineering standards, the difficulty in writing job scheduling/memory management infrastructure and possibly the fact that closed firmware is the norm.
  - alexbaden 2y ago
    Arguably the nvidia AI moat is PyTorch and the heavily optimized libraries behind it. The CUDA language and toolchain helped get that effort off the ground, no doubt, but PyTorch is written and optimized for CUDA first. All other backends work best with similar semantics to CUDA and have to match Cuda semantics to keep their users happy.
  - casperb 2y ago
    Yes he has. I have seen multiple episodes on his YouTube[1] where he absolutely grills the whole company. He also gave them a deadline to opensource the drivers or he would stop trying to make AMD stuff work. Sorry for no direct link, but he has so many and very long videos that it is hard to find the exact spot. https://www.youtube.com/@geohotarchive https://www.youtube.com/@geohotarchive
anthonix1 2y ago
Seems to be an issue on their side. E.g., for a step of GPT2 training on a 7900 XTX [1]: tinygrad is ~440ms, PyTorch 2.4.0.dev20240513 is ~97ms, Karpathy's llm.c with ROCm is ~79ms, and llm.c with custom kernels is ~58ms [1] https://github.com/anthonix/llm.c https://github.com/anthonix/llm.c [2] https://github.com/tinygrad/tinygrad/issues/4301 https://github.com/tinygrad/tinygrad/issues/4301
- xiphias2 2y ago
  That issue seems a month old, while the 58ms number looks 1 day old. I have seen last month getting a lot of work done in improving performance (it's in the release announcement as well), but of course I still don't think it can compete with that number...still, a new comparision would be cool.
  - anthonix1 2y ago
    Ran tinygrad again about a week ago, no change. And still no comment on the issue, will re-run if there is any comment.
    - xiphias2 2y ago
      Thanks for the answer
Lockal 2y ago
Tinygrad targets consumer hardware (to be precise, only Radeon 7900XTX and nothing else[1]), while ROCm does not actually provide good support for such hardware. For example, last release of hipBLASLt-6.1.1 library has deep integration with PyTorch[1], while working only on AMD Instinct hardware. And even for the professional hardware out there, the support period is ridiculous: AMD Instinct MI100 (2020) is not supported. Only 4 years and tens of thousands of dollars worth of hardware is going to the trash, yay! And to be more precise, they still use some core libraries from ROCm stack[3], they just don't use all these fancy multi-gigabyte[4] hardware-limited rocBLAS/hipBLASlt/rocWMMA/rocRAND/etc. libraries. [1] https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox [2] https://github.com/pytorch/pytorch/issues/119081 https://github.com/pytorch/pytorch/issues/119081 [3] https://github.com/tinygrad/tinygrad/blob/v0.9.0/tinygrad/runtime/autogen/hsa.py#L388 https://github.com/tinygrad/tinygrad/blob/v0.9.0/tinygrad/ru... [4] https://repo.radeon.com/rocm/yum/6.1.1/main/ https://repo.radeon.com/rocm/yum/6.1.1/main/
- anthonix1 2y ago
  Even without hipBLASlt, PyTorch is still ~4x faster than tinygrad on a 7900 XTX for GPT2, and works fine. Any idea why?
  - Lockal 2y ago
    Maybe due to 2x slower matmul (well known bottleneck), as one of multiple factors? https://github.com/tinygrad/tinygrad/blob/v0.9.0/extra/gemm/hip_matmul.py https://github.com/tinygrad/tinygrad/blob/v0.9.0/extra/gemm/... There are multiple bounties just for it in https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/edit#gid=0 https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic...
    - anthonix1 2y ago
      I think the matmul issue is symptomatic of a much deeper issue. It would be nice to see less whining and blaming AMD (PyTorch and llm.c actually work on 7900 XTX, and blow tiny grad out of the water in terms of perf!), and more just getting stuff to work.
deleted 2y ago
[deleted]

I see they have experimental AMD backends that don’t use ROCm. Is ROCm that bad that they wrote their own, or was there some other justification for this?

by jarbus 2y ago

I see they have experimental AMD backends that don’t use ROCm. Is ROCm that bad that they wrote their own, or was there some other justification for this?

blihp 2y ago

The author of the library has done numerous YouTube rants on how bad he thinks the AMD compute drivers are.

JMiao 2y ago
What does he get wrong?
- KaoruAoiShiho 2y ago
  He's not wrong.
roenxi 2y ago
Has he? The ones I'm aware of he was complaining about the low quality of the generic kernel drivers. AMD software had a tendency to crash when doing anything outside of standard video games (which was my experience too, but I've caved and bought Nvidia since then; average driver quality of Nvidia on linux seems to be much lower but the kernel doesn't go down which is nice. Got a lot of OOM errors where on ROCm the kernel froze requiring a full system restart). But this is interesting and probably strong evidence that the CUDA API isn't the moat people thought it was. CUDA multiplies matricies and that is close to a commodity operation. The moat actually seems to be Nvidia's higher generic software engineering standards, the difficulty in writing job scheduling/memory management infrastructure and possibly the fact that closed firmware is the norm.
- alexbaden 2y ago
  Arguably the nvidia AI moat is PyTorch and the heavily optimized libraries behind it. The CUDA language and toolchain helped get that effort off the ground, no doubt, but PyTorch is written and optimized for CUDA first. All other backends work best with similar semantics to CUDA and have to match Cuda semantics to keep their users happy.
- casperb 2y ago
  Yes he has. I have seen multiple episodes on his YouTube[1] where he absolutely grills the whole company. He also gave them a deadline to opensource the drivers or he would stop trying to make AMD stuff work. Sorry for no direct link, but he has so many and very long videos that it is hard to find the exact spot. https://www.youtube.com/@geohotarchive https://www.youtube.com/@geohotarchive

anthonix1 2y ago

Seems to be an issue on their side. E.g., for a step of GPT2 training on a 7900 XTX [1]: tinygrad is ~440ms, PyTorch 2.4.0.dev20240513 is ~97ms, Karpathy's llm.c with ROCm is ~79ms, and llm.c with custom kernels is ~58ms [1] https://github.com/anthonix/llm.c https://github.com/anthonix/llm.c [2] https://github.com/tinygrad/tinygrad/issues/4301 https://github.com/tinygrad/tinygrad/issues/4301

xiphias2 2y ago
That issue seems a month old, while the 58ms number looks 1 day old. I have seen last month getting a lot of work done in improving performance (it's in the release announcement as well), but of course I still don't think it can compete with that number...still, a new comparision would be cool.
- anthonix1 2y ago
  Ran tinygrad again about a week ago, no change. And still no comment on the issue, will re-run if there is any comment.
  - xiphias2 2y ago
    Thanks for the answer

Lockal 2y ago

Tinygrad targets consumer hardware (to be precise, only Radeon 7900XTX and nothing else[1]), while ROCm does not actually provide good support for such hardware. For example, last release of hipBLASLt-6.1.1 library has deep integration with PyTorch[1], while working only on AMD Instinct hardware. And even for the professional hardware out there, the support period is ridiculous: AMD Instinct MI100 (2020) is not supported. Only 4 years and tens of thousands of dollars worth of hardware is going to the trash, yay! And to be more precise, they still use some core libraries from ROCm stack[3], they just don't use all these fancy multi-gigabyte[4] hardware-limited rocBLAS/hipBLASlt/rocWMMA/rocRAND/etc. libraries. [1] https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox [2] https://github.com/pytorch/pytorch/issues/119081 https://github.com/pytorch/pytorch/issues/119081 [3] https://github.com/tinygrad/tinygrad/blob/v0.9.0/tinygrad/runtime/autogen/hsa.py#L388 https://github.com/tinygrad/tinygrad/blob/v0.9.0/tinygrad/ru... [4] https://repo.radeon.com/rocm/yum/6.1.1/main/ https://repo.radeon.com/rocm/yum/6.1.1/main/

anthonix1 2y ago
Even without hipBLASlt, PyTorch is still ~4x faster than tinygrad on a 7900 XTX for GPT2, and works fine. Any idea why?
- Lockal 2y ago
  Maybe due to 2x slower matmul (well known bottleneck), as one of multiple factors? https://github.com/tinygrad/tinygrad/blob/v0.9.0/extra/gemm/hip_matmul.py https://github.com/tinygrad/tinygrad/blob/v0.9.0/extra/gemm/... There are multiple bounties just for it in https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/edit#gid=0 https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic...
  - anthonix1 2y ago
    I think the matmul issue is symptomatic of a much deeper issue. It would be nice to see less whining and blaming AMD (PyTorch and llm.c actually work on 7900 XTX, and blow tiny grad out of the water in terms of perf!), and more just getting stuff to work.

deleted 2y ago

[deleted]