7 ms·
Aiter: AI Tensor Engine for ROCm
- carbocation 2y agoI'm not a python expert, but this feels very odd to me (both the *init* construction and the return [tgemm.mm](http://tgemm.mm/ http://tgemm.mm/)(input, self.weight, self.bias, None, None) call, which looks like markdown to me: from aiter.tuned_gemm import tgemm import torch class LinearLayer(torch.nn.Module): def **init**(self, in_features, out_features): super(LinearLayer, self).**init**() self.weight = torch.nn.Parameter(torch.randn(out_features, in_features).cuda()) self.bias = torch.nn.Parameter(torch.randn(out_features).cuda()) def forward(self, input): input = input.cuda() return [tgemm.mm](http://tgemm.mm/)(input, self.weight, self.bias, None, None)
- evertedsphere 2y agoyep the syntax highlighting / doc hyperlinking clearly broke there (or, less charitably, whatever llm produced that prose had a moment) it's __init__ of course
- deleted 2y ago[deleted]
- fock 2y agoalso why is it calling .cuda() to move tensors to a cuda driver? I suppose this is because this is based on HIP - which comes with it's own set of problems, but that's ROCm for the masses I guess. Also the tgemm.mm has to be a torch module (at first I thought this was some lowlevel library which they now have a preview of, because there is a ROCm-torch already ...) which is evident from the table just before the summary. That table also smells like they are mostly focused on inference... EDIT: seems official ROCm-torch is also based on HIP.
- Lerc 2y agoI was puzzling over the code wondering why they .cuda() everything like that when I realised that that was only the beginning of the weirdness. I'm assuming the scrambled annotations were due to some odd chain of things the code went through on the way to becoming a post. Maybe they did it as a parable about the problems of having many layers of abstraction causing processes with unintended consequences?
- cavisne 2y agoYeah this is AMD in a nutshell. A bunch of fluffy descriptions and then the only concrete example would clearly never run. EDIT: They fixed the code pretty quickly
- Tpt 2y agoIf I understand correctly, this library provides some Torch kernels customized for AMD hardware. Why haven't they just upstreamed them to PyTorch for better adoption? Also, they seem to demo usage with Torch default eager execution mode and not Torch JIT/TorchScript. Is this library compatible with TorchScript?
- barrenko 2y agoI really do not understand why can't they just work with existing OSS developers pulling their hair out trying to make AMD devices work and instead do it this way. It's like Mozilla with the questionable decisions.
- hoomanmo 2y agowhich Mozilla's questionable decisions are you referring to?
- fock 2y agoI think they are taken over by exactly the same people leading the AI-hype. Funny how in this article they are a) not advertising clearly what they are doing, b) solving a small subset of problems in a way noone asked for (I think most people just want ROCm to work at all...) and c) just adding to a complex product without any consideration of actually integrating with its environment. I guess it's vibecoding "AI"...
- fc417fc802 2y ago> I think most people just want ROCm to work at all I think most people don't want to have to think about vendor lock-in related bullshit. Most people just want their model to run on whatever hardware they happen to have available, don't want to have to worry about whether or not future hardware purchases will be compatible, and don't want to have to rewrite everything in a different framework. Most people fundamentally don't care about ROCm or CUDA or OneAPI or whatever else beyond a means to an end.
- microtonal 2y ago
- dailykoder 2y agoStill waiting for ROCm on my cheap Radeon RX 7600. Would be nice to play around with it a little. I know that this card is nothing fancy. There is somewhere a github issue where they announced to port it for linux to consumer cards, but last time I checked (a few days ago) it still wasn't available
- oynqr 2y agoI used rocm on an RX 7600 a month after launch. Having no official support does not at all mean it doesn't work.
- Rounin 2y agoYou should be able to make it think you have another card: export HSA_OVERRIDE_GFX_VERSION=10.3.0 The possible values are said to be: # gfx1030 = "10.3.0" # gfx900 = "9.0.0" # gfx906 = "9.0.6" # gfx908 = "9.0.8" # gfx90a = "9.0.a"
- ColonelPhantom 2y agoI believe the override for GP's 7600 is 1100 or 11.0.0 as GFX1030 is RDNA2 (6800 XT).
- TheCycoONE 2y agoThe 7900 models are all 1100, the 7800XT is 1101 and the 7600 is 1102. See Shader ISA: https://www.techpowerup.com/gpu-specs/radeon-rx-7600-xt.c4190 https://www.techpowerup.com/gpu-specs/radeon-rx-7600-xt.c419...
- slavik81 2y agoTelling ROCm to pretend that your RDNA 3 GPU (gfx1102) is an RDNA 2 GPU (gfx1030) is not going to work. The ISAs are not backwards-compatible like that. You might get away with pretending your gfx1102 GPU is a gfx1100 GPU, but even that depends on the code that you're loading not using any gfx1100-specific features. I would generally recommend against using this override at all for RDNA 3 as those ISAs are all slightly different. In any case, the possible values can be found in the LLVM documentation [1]. I would recommend looking closely at the notes for the generic ISAs, as they highlight the differences between the ISAs (which is important when you're loading code built for one ISA onto a GPU that implements a different ISA). [1]: https://llvm.org/docs/AMDGPUUsage.html#processors https://llvm.org/docs/AMDGPUUsage.html#processors
- yu3zhou4 2y agoReally interesting, how it compares to tinygrad support for AMD GPUs?
- fancyfredbot 2y agoWow, it sure sounds like a mess under there. They used 4 different languages? Using one high level language and assembly sounds fine, but four feels incoherent. Would love to know why this has had happened. "This infrastructure is built upon a variety of underlying technologies, including Triton, CK (Compute Kernel), ASM (Assembly), and HIP (Heterogeneous Interface for Portability)."
- jfim 2y agoThat's not exactly unusual, for example pytorch has Python, C++, C, and Cuda.
- fancyfredbot 2y agoNotice those are all (except arguably CUDA) very mainstream languages. All four of AMDs are niche. Upstreaming this into pytorch would double the number of languages used. (Although HIP is very similar to CUDA)
- ColonelPhantom 2y agoI believe that PyTorch already uses Triton; I recently tried to do torch.compile on a Windows machine and it did not work because the inductor backend relies on Triton.
- atq2119 2y agoHIP is essentially the same as CUDA, CK is not a language but a library, and assembly is basically used in the Nvidia ecosystem as well, in the form of PTX. There is absolutely nothing out of the ordinary here. Yes, it's multiple languages, but not any more or any different than what you'd use on an Nvidia platform (except obviously for the assembly part -- AMD's ISA is different from PTX, but that's to be expected).
- fancyfredbot 2y agoI agree using both a high level and a low level language is normal, and yes using libraries is fine. It's having both Triton and HIP in the same project which I find weird. It feels very fragmented to me to use two high level languages. Maybe it makes sense given Triton is easier to use but less fully featured, but it definitely didn't strike me as normal. I would be interested to know if NVIDIA use more than CUDA and PTX/SASS to write CUDNN and CUBLAS.
- fragebogen 2y agoSilly question perhaps, but is this a true CUDA equivalent? Why (not)?
- moralestapia 2y agoThis is equivalent to something like cuDNN, a CUDA library. Aiter is a ROCm library. ROCm is the thing that is like CUDA, but for AMD.
- bayindirh 2y agoI just want to remind everyone that El Capitan, Frontier and LUMI supercomputers are powered by AMD instinct cards. El Capitan is #1 in TOP500. Frontier is #2, LUMI is #8. ROCm development is probably mainly driven by the needs of these supercompuers' users currently. So, we're seeing the tip of the iceberg. Also ROCm packages continue to land on Debian, so there's more than meets the eye. Note: Search "AMD Instinct" at https://top500.org/lists/top500/list/2024/11/ https://top500.org/lists/top500/list/2024/11/. There are way more systems.
- brrrrrm 2y agoDo super computers run in fp64 mostly? At fp8 an h100 hits 2 petaflops, and with only 1000 of them you’ve got more compute power than el capitan (in raw flop count)
- dhruvdh 2y agoEl Capitan can also do FP8. HPC requires double precision generally but people are trying to make low precision work.
- touisteur 2y agoI'm particularly fond of the Ozaki scheme https://arxiv.org/html/2306.11975v4 https://arxiv.org/html/2306.11975v4 and its recent refinements. Hopefully it trickles down to standard HPC libraries soon.
- bayindirh 2y agoDisclosure: I'm an HPC admin who developed a materials simulation framework for my Ph.D. Simulations run on FP64, and you have to since you're already approximating stuff with numerical algorithms (analytic solution of many things are impossible anyway). Even if you can do things with FP8, transferring everything to GPU is not trivially possible. A simulation contains tons of different algorithms, and not all of them can be modeled as a set of matrix operations effectively. Also, moving kernels in an out of GPU is not an instant affair, plus moving data to GPU is always more expensive. You have GPUDirect and MultiDMA engines in modern GPUs, but they need hardcore coding and knowing what you're doing if you're not solving popular stuff with established libraries and so on. Plus, if you don't prefer to be vendor locked, at least one of the vendors artificially limit the performance you can get from their cards. On the other hand, all of the prominent linear algebra libraries squeeze out the CPUs you have relatively easily, and you don't have to have matrices and vectors to get this performance from CPUs anyway. Lastly, I want to touch on that parallelization such problems are not always trivial even on CPUs. When you go multinode via MPI, things get fun. Getting GPUs into that mix is somewhat of a madness if you're not prepared.
- ursulasorensen 2y ago[dead]
- mjburgess 2y agoAny one try any of this on a few 7900xtx (or familiarity with this hardware and platform)? I've just purchased 6 for some small-scale experimentation. I'm thinking the next machine I'll use AMD Radeon PRO W7900 (to get 128 GB VRAM / machine).
- almostgotcaught 2y agoJust export HSA_OVERRIDE_GFX_VERSION=11.0.0 and things should mostly work. Off the top of my head, some of the fp8 types aren't supported but <shrug>
- mjburgess 2y agoThanks -- I don't need everything to work, just enough to explore the platform and develop some realistic prototypes which can be moved on to probably the Radeon PROs.
- almostgotcaught 2y agoI run a large test suite daily (~30000) meant for MI300 on my local 7900. I don't keep track of fails outside of a specific few tests that I'm interested in but in general I get about 70-80% passing.
- slavik81 2y agoThe RX 7900 XTX and Radeon PRO W7900 are already 11.0.0. That override is unnecessary.
- fngarrett 2y agoWe have a dual W7800 system in-house as our `gfx1100` rig. I'll try to install and run through the tests sometime this week.
- manjunaths 2y agoI have a 7900 GRE, which is the same except less memory. I run Gemma 3, LLama 3.1, the QwQ models and the DeepSeek distilled models using llama.cpp. They run fine, I especially like the new Gemma3-27b-Q6 (20 GB model), I get 2 tok/s on it. I have also run Hunyuan3d-2 and generated 3d models. You would've to separate out the model generation and texture generation phase, but it works. I run ComfyUI and bootleg gguf models. This is all on windows. Now even WSL2 works, so I am using Ubuntu-24.04 on Windows 11 to run Hunyuan3D-2. For LLMs, llama.cpp native binaries are available. Everything just works out of the box.
- cavisne 2y agoSo to do an efficient MM on AMD you need to find every MM in the pytorch model and replace it with a call to this library? Seems like something that should've been fixed years ago. Also I assume nvidia does the same thing but it is still hilarious that this is how it works https://github.com/ROCm/aiter/blob/main/aiter/configs/bf16_tuned_batched_gemm.csv https://github.com/ROCm/aiter/blob/main/aiter/configs/bf16_t...
- randomNumber7 2y agoWhy is everyone using the GPUs of this other company for AI?
- WhitneyLand 2y agoPerformance increased 100% on an MI300X running a large LLM. On one hand, cool. On the other hand wow have they been leaving a lot of performance on the table. How does the performance compare to NVidia now?