7 ms·
Why CUDA translation wont unlock AMD
- pixelpoet 11mo agoActual article title says "won't"; wont is a word meaning habit or proclivity.
- InvisGhost 11mo agoIn situations like this, I try to focus on whether the other person understood what was being communicated rather than splitting hairs. In this case, I don't think anyone would be confused.
- philipallstar 11mo agoProbably best to just fix the spelling.
- Eliovp 11mo agoThat's what you get when you don't use AI to write an article :p
- measurablefunc 11mo ago[flagged]
- imtringued 11mo agoThe AI needs a mental model of the hardware for that to work.
- measurablefunc 11mo agoAlgorithms do not have mental models of anything.
- jsheard 11mo agoThe article is literally about how rote translation of CUDA code to AMD hardware will always give sub-par performance. Even if you wrangled an AI into doing the grunt work for you, porting heavily-NV-tuned code to not-NV-hardware would still be losing strategy.
- measurablefunc 11mo agoThe point of AI is that it is not a rote translation & 1:1 mapping.
- jsheard 11mo ago> Take the ROCm specification, take your CUDA codebase, let one of the agentic AIs translate it all into ROCm ...sounds like asking for a 1:1 mapping to me. If you meant asking the AI to transmute the code from NV-optimal to AMD-optimal as it goes along, you could certainly try doing that, but the idea is nothing more than AI fanfic until someone shows it actually working.
- measurablefunc 11mo agoNow that I have clarified the point about AI optimizing the code from CUDA to fit AMD's runtime what is your contention about the possibility of such a translation?
- bigyabai 11mo agoThere is an old programmer's joke about writing abstractions and expecting zero-cost.
- measurablefunc 11mo agoHow does that apply in this case? The whole point is that the agentic AI/AGI skips all the abstractions & writes optimized low-level code for each GPU vendor from a high-level specification. There are no abstractions other than whatever specifications GPU vendors provide for their hardware which are fed into the agentic AI/AGI to do the necessary work of creating low-level & optimized code for specific tasks.
- cbarrick 11mo agoHas this been done successfully at scale? There's a lot of handwaving in this "just use AI" approach. You have to figure out a way to guarantee correctness.
- measurablefunc 11mo agoThere are tons of test suites so if the tests pass then that provides a reasonable guarantee of correctness. Although it would be nice if there was also proof of correctness for the compilation from CUDA to AMD.
- bigyabai 11mo agoBecause it doesn't work like that. TFA is an explanation of how GPU architecture dictates the featureset that is feasibly attainable at runtime. Throwing more software at the problem would not enable direct competition with CUDA.
- measurablefunc 11mo agoI am assuming that is all part of the specification that the agentic AI is working with & since AGI is right around the corner I think this is a simple enough problem that can be solved with AI.
- Blackthorn 11mo agoI don't know why you're being downvoted because even if you're Not Even Wrong, that's exactly the sort of thing that has been endlessly presented by people trying to sell AI as something that AI will absolutely do for us.
- measurablefunc 11mo ago[flagged]
- bigyabai 11mo agoIt's hard to catch-on to a deliberately dishonest pretense. You could clone 10,000 John Carmacks to do the job for you, Nvidia would still be a $5 trillion business next time you wake up.
- measurablefunc 11mo ago[flagged]
- bigyabai 11mo agoI'm not talking to them. I am responding to you - your sardonic piss-take is against HN guidelines and written in bad-faith.
- measurablefunc 11mo ago[flagged]
- bigyabai 11mo agoSure, and thieves probably recommend that the cops move on & refrain from following where they're headed. Be honest and you won't have to fend-off accusations of bad-faith. I'm inclined to agree with your overall point of AI being overhyped, but you've gutted your own logic so hard in the process that your stance is unrecognizable. You've developed a meaningfully ambiguous stance to an elaborate and deeply incorrect series of arguments.
- j16sdiz 11mo agoThe same as "Why just outsourcing it to <some country >" AI aint magic. You need more effort to manage, test and validate that.
- measurablefunc 11mo ago[flagged]
- j16sdiz 11mo agoSo, your strategy for solving this is: Convert it to another harder problem (AGI). Now it is somebody else (AI researcher)'s problem. This is outsourcing the task to AI researchers.
- measurablefunc 11mo agoThey keep promising that this kind of capability is right around the corner & they keep showing how awesome they are at passing math exams so why is this a more difficult problem than solving problems in abstract algebra & scheme theory on humanity's last exam or whatever is the latest & greatest benchmark for mathematical capabilities?
- Daedren 11mo agoThey all have to make promises and have to dream big to keep the AI bubble from popping.
- measurablefunc 11mo agoI agree which is why it's a bit odd that so many people still think that Sam Altman & Elon Musk are honest technologists instead of unscrupulous grifters.
- j16sdiz 11mo agoI am not saying this is impossible, but I am down voting this because this is _not an interesting discussion_. The whole point of having an online discussion forum is to exchange and create new ideas. What you are advocating is essentially "maybe we can stop generating new ideas because we don't have to. we should just sit and wait"... Well, yes, no, maybe. but this is not what I expect to get from here.
- bee_rider 11mo agoThe AI is too busy making Ghibli profile pictures or whatever the thing is now. We asked it to make a plan for how to fix the situation, but it got stuck. “Ok, I’m helping the people build an AI to translate NVIDIA codes to AMD” “I don’t have enough resources” “Simple, I’ll just use AMD chips to run an AI code translator, they are under-utilized. I’ll make a step by step process to do so” “Step 1: get code kernels for the AMD chips” And so on.
- measurablefunc 11mo agoThe real question is whether it will be as unprofitable to do this type of automated runtime translation from one GPU vendor to another as it is to generate Mario clips & Ghibli images.
- colonCapitalDee 11mo agoNo. This is far beyond the capabilities of current AI, and will remain so for the foreseeable future. You could let your model of choice churn on this for months, and you will not get anywhere. It will be able to reach a somewhat working solution quickly, but it will soon reach a point where for every issue it fixes, it introduces one or more issues or regressions. LLMs are simply not capable of scaffolding complexity like a human, and lack the clarity and rigorousness of thought required to execute an *extremely* ambitious project like performant CUDA to ROCm translation.
- measurablefunc 11mo ago[flagged]
- colonCapitalDee 11mo agoWell that's your problem. Here's a tip: just because someone says something doesn't mean you have to listen to them
- measurablefunc 11mo ago[flagged]
- bigyabai 11mo agoThis explains everything.
- impossiblefork 11mo agoI don't think it really is, especially not if it's turned into a system, with multiple prompts, verification, etc. Humans have problems with IMO problems, and this kind of kernel translation is a problem which is easier to humans, where there's more probably actually more data and a problem where the system can get feedback by simply running it and measuring memory use, runtime etc. It'd be a system and no one has developed it, but I think it can be done with present LLMs as a core mechanism. They just need to be trained with RL on this specific problem. Anyone with a good LLM, from Google to Mistral could probably do this, but it'd be a project.
- outside1234 11mo agoAre the hyperscalers really using CUDA? This is what really matters. We know Google isn't. Are AWS and Azure for their hosting of OpenAI models et al?
- bigyabai 11mo ago> We know Google isn't. Google isn't internally, so far as we know. Google's hyperscaler products have long offered CUDA options, since the demand isn't limited to AI/tensor applications that cannibalize TPU's value prop: https://cloud.google.com/nvidia https://cloud.google.com/nvidia
- wmf 11mo agoAll Nvidia GPUs, which are probably >70% of the market, use CUDA.
- lvl155 11mo agoLet’s just say what it is: devs are too constrained to jump ship right now. It’s a massive land grab and you are not going to spend time tinkering with CUDA alternatives when even a six-month delay can basically kill your company/organization. Google and Apple are two companies with enough resources to do it. Google isn’t because they’re keeping it proprietary to their cloud. Apple still have their heads stuck in sand barely capable of fixing Siri.
- stingraycharles 11mo agoGoogle has their own TPUs so they don’t have any vendor lock-in issues at all. OpenAI OTOH is big enough that the vendor lock-in is actually hurting them, and them making that massive deal with AMD may finally push the needle for AMD and improve things in the ecosystem to make AMD a smooth experience.
- apfsx 11mo agoGoogle’s TPU’s are not powering Gemini or whatever X equivalent LLM you want to compare to.
- rdudek 11mo agoWhat is powering Gemini?
- jsolson 11mo agoTPUs: https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf https://storage.googleapis.com/deepmind-media/Model-Cards/Ge...
- rfw300 11mo agoThis isn't true. Gemini is trained and run almost entirely on TPUs. Anthropic also uses TPUs for inference, see, e.g., https://www.anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services https://www.anthropic.com/news/expanding-our-use-of-google-c... and https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-.... OpenAI also uses TPUs for inference at least in some measure: https://x.com/amir/status/1938692182787137738?t=9QNb0hfaQShWoB65AeyWvw&s=19 https://x.com/amir/status/1938692182787137738?t=9QNb0hfaQShW....
- kj4ips 11mo agoI agree pretty strongly. A translation layer like this is making an intentional trade: Giving up performance and HW alignment for less lead time and effort to make a proper port.
- martinald 11mo agoPerhaps I'm misunderstanding the market dynamics; but isn't AMDs real opp inference over research? Training etc still happens on NVDA but inference is somewhat easy to do on vLLM et al with a true ROCm backend with little effort?
- mandevil 11mo agoYeah, ROCm focused code will always beat generic code compiled down. But this is a really difficult game to win. For example, Deepseek R-1 released optimized for running on Nvidia HW, and needed some adaption to run as well on ROCm. This was for the exact same reasons that ROCm code will beat generic code compiled into ROCm, in the same way. Basically the Deepseek team, for their own purposes, created R-1 to fit Nvidia's way of doing things (because Nvidia is market-dominant) on their own. Once they released, someone like Elio or AMD would have to do the work of adapting the code to run best on ROCm. For more established players who weren't out-of-left-field surprises like Deepseek, e.g. Meta's Llama series, mostly coordinate with AMD ahead of release day, but I suspect that AMD still has to pay for the engineering work themselves while Meta does the work to make it run on Nvidia themselves. This simple fact, that every researcher makes their stuff work on CUDA themselves, but AMD or someone like Elio has to do the work to move it over to get it to be as performant on ROCm, that is what keeps people in the CUDA universe.
- latchkey 11mo agoKimi is the latest model that isn't running correctly on AMD. Apparently close to Deepseek in design, but different enough that it just doesn't work. It isn't just the model, it is the engine to run it. From what I understand this model works with sglang, but not with vLLM.
- suprjami 11mo agoThis is normal. An inference engine needs support for a model's particular implementation of the transformer architecture. This has been true for almost every model release since we got local weights. Really good model providers send a launch-day patch to llama.cpp and vllm to make sure people can run their model instantly.
- latchkey 11mo agoIt isn't about normal or not. It is that those patches are done for Nvidia, but not AMD. It is that it takes time and energy to vet them and merge them into those projects. Kimi has been out for 3 months now and it still doesn't run out of the box on vLLM on AMD, but it works just fine with Nvidia.
- doctorpangloss 11mo agoAll they have to do is release air cooled 96GB GDDR7 PCIe5 boards with 4x Infinity Link, and charge $1,900 for it.
- jmward01 11mo agoRight now we need diversity in the ecosystem. AMD is finally getting mature and hopefully that will lead to them truly getting a second, strong, opinion into ecosystem. The friction this article talks about is needed to push new ideas.
- manjose2018 11mo agohttps://geohot.github.io//blog/jekyll/update/2025/03/08/AMD-YOLO.html https://geohot.github.io//blog/jekyll/update/2025/03/08/AMD-... https://tinygrad.org/ https://tinygrad.org/ is the only viable alternative to CUDA that I have seen popup in the past few years.
- bigyabai 11mo agoViable how? "Feasible" might be a better word here, I haven't heard many (any?) war-stories about a TinyBox in production but maybe I'm OOTL.
- erichocean 11mo agoBoth Mojo and ThunderKittens/HipKittens are viable on AMD.
- balaclava9 11mo agoMojo runs faster on nvidia hardware than CUDA in some cases. https://x.com/clattner_llvm/status/1982196673771139466?s=61 https://x.com/clattner_llvm/status/1982196673771139466?s=61
- djsjajah 11mo agoI can't tell if you are making a joke or not. They are not even remotely equivalent. tinygrad is a toy. If you are serious, I would be interested to hear how you see tinygrad replacing CUDA. I could see a tiny grad zealot arguing that it is gong to replace torch, but CUDA?? Have you looked into AMD support in torch? I would wager that like for like, a torch/amd implementation of a models is going to run rings around a tinygrad/amd implementation.
- buggyworld 11mo ago[flagged]
- KetoManx64 11mo agoWhich LLM did you use to write this?
- throwaway31131 11mo agoI don't think this was the point of the post at all. Their bottom line summed it up perfectly. "We’re not saying “never use CUDA-on-AMD compilers or CUDA-to-HIP translators”. We’re saying don’t judge AMD based on them."
- fulafel 11mo agoVulkan Compute is catching up with HIP (or whatever the compatibility stuff is called now), which seems like a welcome break from CUDA - in this benchmark it beats CUDA in some benchmarks on AMD: https://www.phoronix.com/review/rocm-71-llama-cpp-vulkan https://www.phoronix.com/review/rocm-71-llama-cpp-vulkan
- pjmlp 11mo agoFor most devs using GLSL instead of C++20, or Python GPU JIT, is a downgrade in developer experience.
- apitman 11mo agoOur open source library is currently hard locked into CUDA due to nvCOMP for gzip decompression (bioinformatics files). What I wouldn't give for an open source implementation, especially if it targeted WebGPU.
- latchkey 11mo agoA bit of background. This is directed towards Spectral Compute (Michael) and https://scale-lang.com/ https://scale-lang.com/. I know both of these guys personally and consider them both good friends, so you have to understand a bit of the background in order to really dive into this. My take on it is fairly well summed up at the bottom of Elio's post. In essence, Elio is taking the view of "we would never use scale-lang for llms because we have a product that is native AMD" and Michael is taking the view of "there is a ton of CUDA code out there that isn't just AI and we can help move those people over to AMD... oh and by the way, we actually do know what we are doing, and we think we have a good chance at making this perform." At the end of the day, both companies (my friends) are trying to make AMD a viable solution in a world dominated by an ever growing monopoly. Stepping back a bit and looking at the larger picture, I feel this is fantastic and want to support both of them in their efforts.
- Eliovp 11mo agoJust to clarify: this post was not written against Spectral Compute. Their recent investment news was the trigger for us to finally write it yes, but the idea has been on our minds for a long time. We actually think solutions like theirs are good for the ecosystem, they make it easier for people to at least try AMD without throwing away their CUDA code. Our point is simply this: if you want top-end performance (big LLMs, specific floating point support, serious throughput/latency), translation alone is not enough. At that point you have to focus on hardware-specific tuning: CDNA kernel shapes, MFMA GEMMs, ROCm-specific attention/TP, KV-cache, etc. That’s the layer we work on: we don’t replace people’s engines, we just push the AMD hardware as hard as it can go.
- jfalcon 11mo agoCUDA isn't all that and a bag of chips. It just is the Facebook/Twitter of the data science and from that LLM space. There are Tensor processors and other ASIC processing for specific compute functions that can give Nvidia a challenge but it's not unknown to every gamer that there has always been a performance difference between Nvidia and AMD/ATI. Ok, point made Nvidia. Kudos. ATI had their moment in the sun before ASIC ate their cryptocurrency lunch. So both still had/have relevance outside gaming. But, I see Intel is starting to take GPU space seriously and they shouldn't be ruled out. And as mentioned elsewhere in the comments, there is Vulkan. There is also this idea of virtualized GPU as now the bottleneck isn't CPU... it's now GPU. As I mentioned there are Tensors, Moore's Law thresholds coming back again with 1 nanometer manufacturing, there is going to be a point where we hit a threshold again with current chips and we will have a change in technology - again. So while Nvidia is living the life - unless they have a crystal ball of how tensors are going to go that they can move CUDA towards, there is going to be a "co-processor" future coming up and with that the next step towards NPUs will be taken. This is where Apple is aligning itself because, after all, they had the money and just said "Nope, we'll license this round out..." AMD isn't out yet. They, along with Intel and others, just need to figure out where the next bottlenecks are and build those toll bridges.
- musicale 11mo agoIf you can run PyTorch well, isn't that good enough for a lot of people?