9 ms·
Open-source project ZLUDA lets CUDA apps run on AMD GPUs
- dpflan 3y agoThis seems relevant for discussion: Nvidia bans using translation layers for CUDA software to run on other chips [1] ____ [1] https://news.ycombinator.com/item?id=39592689 https://news.ycombinator.com/item?id=39592689
- dotnet00 3y agoShould be emphasized again that contrary to the article's claim, the clause in question has been in CUDA's EULA, even in the downloads (contrary to the updated statement in the article), since Jan 2022.
- croes 3y agoMeans Nvidia is even longer anti competitive than we thought.
- contravariant 3y agoWait, you thought they started after 2022?
- mort96 3y agoDoes that even matter? It's not like you need someone's permission to implement a system with a compatible interface to another. It violates the EULA but you don't need to accept the EULA unless you download the CUDA software, which I guess the authors of ZLUDA could avoid doing
- jsheard 3y agoThe complication is that most CUDA apps you would want to run on ZLUDA contain first-party libraries provided by Nvidia (e.g. cuDNN) which may have restrictive license terms saying you're not allowed to run them on a third-party runtime. ZLUDA itself may be legally in the clear as a cleanroom reimplementation free of Nvidia code, but it's not so clear-cut for the users of ZLUDA.
- mort96 3y agoAha, so we'd need clean-room re-implementations of those libraries too, in principle.
- easyThrowaway 3y agoWhat's the difference between this and Wine/Proton? I guess Microsoft EULA has similar conditions, if they were enforceable wouldn't Microsoft do the same and send a C&D to Wine devs?
- parentheses 3y agoEULA permits the companies to have grounds to take someone to court. Their choice to prosecute. It's a reserved right more so than one that they use 100%. In this case translating CUDA can allow AMD to chip away at NVidia's market share.
- adastra22 3y agoNot if the person being sued never “signed” the EULA.
- deleted 3y ago[deleted]
- LoganDark 3y agoNvidia cares about the people reverse-engineering CUDA, who don't have anything to reverse-engineer unless they actually download the SDK. Of course, when you download the SDK it has an associated license.
- TehCorwiz 3y agoIt's also possible to do a clean-room reverse where one group tests the hardware and software and writes specifications and another group who has never seen the hardware or docs then does the implementation. This has been legally tested and protected going back to at least the early 1980s.
- mandevil 3y agoIt is possible, but a) it is expensive as hell to get enough engineers whom you can prove in court have no exposure to the original software (most SWE's would get some exposure just naturally in college nowadays, leave alone at any sort of job) and b) CUDA is constantly changing and updating and so you need to have this expensive clean-room process going constantly or else you will fall behind. The most famous case of clean-room reverse engineering is for the original BIOS chips back in the early 1980's, where the chips themselves couldn't change- they were hardware! It's going to be orders of magnitude more difficult to do that for software packages that change regularly.
- adastra22 3y agoNVIDIA doesn’t have the authority to do that. There’s no NVIDIA SDK involved here.
- TehCorwiz 3y agoIf I'm not using Nvidia hardware, and I don't use Nvidia drivers, and I haven't agreed to their EULA then why would I care? Emulation is legally protected both explicitly and through legal precedence. The replication of APIs for compatibility purposes has been argued to the US Supreme Court and found to not be copyrightable. At least within some pretty broad scope. IANAL, but I fail to see what legal basis Nvidia is relying on. For a single user or company who owns no Nvidia hardware this feels moot. For a company with existing Nvidia hardware I could see them having an argument, kinda. But wouldn't that be squarely in the anti-competitive behavior wheelhouse?
- AndrewKemendo 3y ago> But wouldn't that be squarely in the anti-competitive behavior wheelhouse? Precisely why they are making that statement. The goal is to threaten people who attempt to avoid CUDA
- jsheard 3y ago> If I'm not using Nvidia hardware, and I don't use Nvidia drivers, and I haven't agreed to their EULA then why would I care? If the CUDA software you want to run on ZLUDA contains first-party Nvidia libraries, which it usually does, you have to care about how those dependencies are licensed.
- TehCorwiz 3y agoYeah, I commented below at someone who mentioned that. I wasn't aware that people were distributing Nvidia binary libraries with their apps.
- Cheer2171 3y ago> If I'm not using Nvidia hardware, and I don't use Nvidia drivers, and I haven't agreed to their EULA then why would I care? So in your entire life, you have never downloaded an Nvidia driver and clicked through the EULA? Once you agree, you've agreed.
- 3y ago
- gsich 3y agoMy hardware - my rules.
- jrepinc 3y agoFsck you nvidia even more. Just going the same evil ways as Nintendo I see. Good thing I don't waste my money on your products.
- mtillman 3y agoI find myself more upset with AMD for completely dropping the ball on firmware and software. Almost intentional levels of incompetence.
- gnabgib 3y ago[flagged]
- dpflan 3y agoI think it is a relevant zeitgeist reaction to NVIDIA banning translation of CUDA to non-NVIDIA hardware post from earlier.
- drakerossman 3y agoIt now supports AMD GPUs since 3 weeks ago, check the latest commit at the repo: https://github.com/vosen/ZLUDA https://github.com/vosen/ZLUDA The article also mentions exactly this fact.
- KeplerBoy 3y agoThat's exactly what the post from 3 weeks ago discussed. AMD dropped ZLUDA funding, the remains were opensourced and it won't ever be a viable option.
- bri3d 3y ago> it won't ever be a viable option For production workloads, I generally agree. It's an unsupported hack with a questionable future, I wouldn't do anything money-making with it. However, for tinkering and consumer workloads, it already works pretty well. Enough of cuDNN and cuBLAS work to run PyTorch and in turn, Stable Diffusion with https://github.com/lshqqytiger/ZLUDA https://github.com/lshqqytiger/ZLUDA - there's even a fairly user-friendly setup process already in https://github.com/vladmandic/automatic https://github.com/vladmandic/automatic . I was able to get a personal non-ML related project working on my AMD card in just a few minutes, which saved me a lot of development time before I then deployed the production workload on NV hardware (this is probably why AMD pulled the plug on the project - it's almost more of a boost to NV than anything else, AMD really need people to be writing code on ROCm to deploy on AMD datacenter hardware).
- Blackthorn 3y agoIt's absolutely absurd that AMD stopped funding this, since right as it got released as open source it started producing value for AMD users. You'd think this exact thing would be their top priority, but instead they've been faffing around for years with two (are they up to three now?) alternate APIs with minimal support so far.
- deleted 3y ago[deleted]
- bri3d 3y agoI don't really think this was in AMD's best interest when you think about it strategically. Unless it were production-grade and legally tested, it's basically a tool that would enable developers to build applications using AMD and then deploy on NV. Perhaps a short term win on the consumer card side, but a long term foot-gun that would only serve to continually entrench NV in the datacenter.
- outworlder 3y agoThat's a problem indeed, since it would further entrench CUDA. If you want people to develop for your platform, it could be counter-productive. That said, what _is_ AMD's platform? OpenCL? Vulkan compute? If they don't have an alternative, then the strategy doesn't make sense.
- incrudible 3y ago> What is AMDs platform? That is a very good question that apparently AMD has no good answer for. I have lost track of the amount of half baked implementations for GPGPU that AMD has attempted and then left to rot. Even if they told me they had the answer now, and they are going to put all their focus on that, I would not trust them to deliver on it. Their best bet is to create implementations for popular libraries like torch that actually stand a chance to work as a drop in replacement.
- 3y ago
- 3abiton 3y agoAnyone managed to get it working for AMD iGPU (APU) yet? I got the vega archi, still no luck running LLMs with either ZLUDA or ROCm backend.
- lhl 3y agoSomeone got ZLUDA running llama.cpp a while back (search the ZLUDA/llama.cpp issues). If I recall, it ran about half the speed of the existing ROCm implementation. I tried ROCm on my iGPU last year and you do get a bit of a benefit for prompt processing (5x faster) but inference is basically bottlenecked by the memory bandwidth whether you're on CPU or GPU. Here were my results: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1041125589 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... Note, only GART memory, not GTT is accessible in most inferencing options, so you will only basically be able to load 7B quantized models into "VRAM" (BIOS controlled, but usually max out at 6-8GB). I have some notes on this here: https://llm-tracker.info/howto/AMD-GPUs#amd-apu https://llm-tracker.info/howto/AMD-GPUs#amd-apu If you plan on running a 7B-13B model locally, getting something like a RTX 3060 16GB (or if you're on Linux, the 7600 XT 16GB might be an option w/ HSA_OVERRIDE) is probably your cheapest realistic option and will give you about 300 GB/s of memory bandwidth and enough memory to run quantizes of 13/14B class models. If you're buying a card specifically for GenAI and not going to dedicate time to fight driver/software issues, I'd recommend going w/ Nvidia options first (they typically actually give you more Tensor TFLOPS/$ as well).
- 3abiton 3y agoThis is an amazing experiment summary! I wonder if your APU conclusion still stands (CPU for bigger models otherwise APU), given all the advancements so far.
- lvl102 3y agoIf AMD couldn’t do it by now they either have no intention or process. The fact that they are selling investors on this notion that they can compete with Nvidia in AI space is borderline fraud.
- Zambyte 3y ago> The fact that they are selling investors on this notion that they can compete with Nvidia in AI space is borderline fraud. Idk, running 7B language models on my 6650 XT with ROCm has been pretty slick. Doesn't seem like fraud to me.
- zero_k 3y agoThis confirms what everyone who ever touched AMD GPGPUs knows -- that the only thing holding back AMD from becoming a 2 Trillion dollar company is their absolutely atrocious software. I remember finding a bug in their OpenCL compiler [1], but crashing their OpenCL compiler via segfault was also a piece of cake (that was never fixed, I gave up on reporting it). AMD not developing a competitor to CUDA was the most short-sighted thing I have ever seen. I have no idea why their board hasn't been sacked and replaced with people who understand that you can make the best hardware out there, but if your SW to use it is -- to be very mild -- atrocious, nobody is gonna buy it or use it. Us, customers, are left to buy the overpriced NVidia cards because AMD's board is too rich to give a damn about a trillion or so of value left on the table. Just... weird. Whoever owns AMD stock I hope is asking questions, because that board needs to go down the nearest drain. [1] https://github.com/msoos/amdmiscompile https://github.com/msoos/amdmiscompile -- they eventually fixed this
- outworlder 3y ago> This confirms what everyone who ever touched AMD GPGPUs knows -- that the only thing holding back AMD from becoming a 2 Trillion dollar company is their absolutely atrocious software. Indeed. On the flip side they are quite more friendly to open-source, in general, compared to NVidia that's actively hostile(and has been for a while(see Linus "F* you!" video). Companies that develop hardware generally suck at software. There are exceptions, but they aren't numerous (and indeed have been rewarded in their stock price). I do not know anything about AMD's company culture in their software business units, but fixing that generally requires pretty large changes. > I have no idea why their board hasn't been sacked and replaced with people who understand that you can make the best hardware out there, but if your SW to use it is -- to be very mild -- atrocious, nobody is gonna buy it or use it. You probably can't just replace the board (unless C-level mandates are the only thing dragging down the company). You need to replace many more management levels, including a sizable portion of middle management. Sometimes even ICs if software hiring hasn't been properly handled.
- littlestymaar 3y ago> Companies that develop hardware generally suck at software. There are exceptions, but they aren't numerous (and indeed have been rewarded in their stock price) True, and you can even get the highest market cap as a hardware manufacturer who suck at software.
- indymike 3y ago> Intel eventually decided there was “no business case for running CUDA applications on Intel GPUs”, Oh, boy.
- joe_the_user 3y agoOne simple way to put things is that at a certain size and age, every company is an aspiring monopolist, not an aspiring competitor.
- winwang 3y agoThat would make more sense in the alternate history where Intel and AMD never try to make GPUs anymore.
- joe_the_user 3y agoNot necessarily. They're making GPUs but they're avoiding any head-to-head competition with NVidia and instead trying for a smaller share of the market but one they have some unique advantage or other.
- asdff 3y agoIntels graphics wing is so bad they had to stop calling it intel hd because of the taste it left in people's mouths.
- Steltek 3y agoHas anyone tried this to run OSS photogrammetry tools like Meshroom? They mention a few proprietary ones in the article but my needs are pretty small.
- singhrac 3y agoAlso relevant is geohot's persistent struggles with (expensive) AMD GPUs: https://twitter.com/__tinygrad__/status/1764734675002810622 https://twitter.com/__tinygrad__/status/1764734675002810622
- parentheses 3y agoThis is almost identical to Oracle vs Google re: using JVM bytecode.
- bri3d 3y agoI don't really think so; what's in dispute isn't the bytecode translation, it's locking the higher-level library IP to hardware. This would be like Google saying "you can only run our Android applications on a Google-approved phone," which my understanding is, they do when it comes to their Play frameworks and things like Maps.
- ddtaylor 3y agoGoogle kind of does that with pixel phones there are features that are only available on their Hardware that you can't run otherwise even if you put the APK over Etc one is holdforme
- deleted 3y ago[deleted]
- v3ss0n 3y agoHe's dead, Jim
- Der_Einzige 3y agoCan anyone confirm if this actually works in practice for GenAI? In general, CUDA translation layers are usually broken for SOTA ML applications.
- dheera 3y agoDoes this work on an AMD 7950X?
- MikeTheRocker 3y agoNo, the 7950X is a CPU. CUDA is an API for computing on GPUs.
- derstander 3y agoTo be fair AMD has a graphics card with the model name 7900XT so the names aren’t that far apart in Levenshtein distance.
- dheera 3y agoYeah, I figured, though it has an iGPU ... would be nice to just test out whether I could run CUDA code on it, even if slow.
- varispeed 3y ago> AMD evaluated ZLUDA for two years, but also decided not to go further with the project – at which point, Janik open-sourced the updated code. Such a dick move from AMD.
- asdff 3y agoTheir legal team probably said the fees from the resulting nvda war would be out of budget
- ok_dad 3y agoIs there a programming language that compiles into any of the various kernel languages like Metal, CUDA, whatever AMD has, etc? If not, why not? We have C compilers that compile to various CPU architectures. Shouldn’t there be a compiler to GPU architectures? Perhaps it’s just that no one has created it yet?
- jawilson2 3y agoDo you count OpenCL? https://www.khronos.org/api/opencl https://www.khronos.org/api/opencl
- ok_dad 3y agoI think so, yes! Even more so because it works with CPU and other things too.
- ortichic 3y agoOpenMP 5 specified GPU support. A quick search suggests that some compilers at least partially support it by now
- Eager 3y agoSomeone should have an LLM start generating random valid CUDA programs. Compile each one to get a binary. Train a language model with the source and output binary. Hey presto, clean room compiler. Edit: Oh wait.. duh.. just train it on the equivalent target source. Presumably you can do this for other targets as well.
- ConanRus 3y ago[dead]
- T3RMINATED 3y ago[dead]
- can16358p 3y agoDoes anyone know would there be support for Apple Silicon GPUs/Metal?
- qwertox 3y agoPrevious discussion 22 days ago: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source [0], 400 comments. Noteworthy top comment in that thread: > This event of release is however a result of AMD stopped funding it per "After two years of development and some deliberation, AMD decided that there is no business case for running CUDA applications on AMD GPUs. One of the terms of my contract with AMD was that if AMD did not find it fit for further development, I could release it. Which brings us to today." from https://github.com/vosen/ZLUDA?tab=readme-ov-file#faq https://github.com/vosen/ZLUDA?tab=readme-ov-file#faq [0] https://news.ycombinator.com/item?id=39344815 https://news.ycombinator.com/item?id=39344815
- thomasahle 3y ago> One of the terms of my contract with AMD was that if AMD did not find it fit for further development, I could release it. I need to put this in all contracts for all jobs in the future.
- dang 3y agoThanks! Macroexpanded: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source - https://news.ycombinator.com/item?id=39344815 https://news.ycombinator.com/item?id=39344815 - Feb 2024 (410 comments) Zluda: Run CUDA code on Intel GPUs, unmodified - https://news.ycombinator.com/item?id=36341211 https://news.ycombinator.com/item?id=36341211 - June 2023 (90 comments) Zluda: CUDA on Intel GPUs - https://news.ycombinator.com/item?id=26262038 https://news.ycombinator.com/item?id=26262038 - Feb 2021 (77 comments) Also recent and related: Nvidia bans using translation layers for CUDA software to run on other chips - https://news.ycombinator.com/item?id=39592689 https://news.ycombinator.com/item?id=39592689 - March 2024 (155 comments)
- physicsguy 3y agoSaid it before and said it again, the issue with AMD GPUs is not individual kernels which are easy to translate, but the libraries. From the release notes saying 'Add minimal support of cuDNN, cuBLAS, cuSPARSE, cuFFT, NCCL, NVML' it looks like this project was going towards this which is great. Whether it'll have momentum after AMD stop funding it... who knows.
- simonw 3y agoI heard an interesting rumor recently that the person responsible for CUDA at NVIDIA spent YEARS fighting for resources and trying to convince the company to take the project seriously. Without CUDA, there's absolutely no way NVIDA would be a nearly trillion dollar company today.