7 ms·

Zluda: Run CUDA code on Intel GPUs, unmodified

237 points by goranmoomin 3y ago

01100011 3y ago
> Is ZLUDA a drop-in replacement for CUDA? Yes, but certain applications use CUDA in ways which make it incompatible with ZLUDA So no then.
- jjallen 3y ago
  yeah would probably be good to include where it doesn't work. 1% of the time? 10%?
  - mattkrause 3y ago
    It's near the bottom: "What is the status of the project? This project is a Proof of Concept. About the only thing that works currently is Geekbench. It's amazingly buggy and incomplete. You should not rely on it for anything serious."
- meepmorp 3y ago
  50% of the time it works every time.
  - catchnear4321 3y ago
    that’s not quite accurate. it works 100% of the time. until it does not.
- glimshe 3y ago
  It's a drop-in replacement in the sense that you don't need to modify your code. But it has limitations/incompatibilities. Contrast to something that isn't a drop-in replacement... That would require changes to the application.
  - 01100011 3y ago
    This statement makes no sense. "It's compatible with CUDA as long as you don't use all the features of CUDA." So it's a drop in replacement for some subset of modern CUDA. I feel like most folks who are upvoting this don't program CUDA professionally or aren't very advanced in their usage of it.
    - dragonwriter 3y ago
      “Drop-in” = to the extent it works, it requires no app changes. “Complete” = it covers everything. It is drop-in, but not complete.
      - dahart 3y ago
        The issue being that in the original question “drop-in” implies complete, that’s what being a drop-in replacement actually means in other contexts. If it’s not complete, then it’s not really drop-in, even though I don’t necessarily disagree with your definition. You can be right, and parent can be right too, IMHO. The FAQ question is stated ambiguously and in misleadingly black and white terms, and the answer really does look kinda funny starting with the word “Yes” and then following that with “but… not exactly”. Wouldn’t it be better to say drop-in is the goal, and because it’s not complete, we’re not there yet?
    - glimshe 3y ago
      Does the statement make no sense or you simply don't like it? I can understand what it says, sounds like plain English to me. I think that what you're trying to say is that before claiming to be a "drop-in replacement", make sure that your supported feature set is representative enough of mainline CUDA development.
YetAnotherNick 3y ago
If a single dev could do it, why can't AMD do the same for their GPUs.
- simcop2387 3y ago
  Patents and software licenses probably.
- mrstone 3y ago
  I imagine they could, but it is probably more of a legal thing.
  - viewtransform 3y ago
    Yes it is a legal issue. AMD cannot implement CUDA. However, they have gone around that by creating HIP which is a CUDA adjacent language that runs on AMD and also translates to CUDA for Nvidia GPUs. There is also the HIPify tool to automatically convert existing sources from CUDA to HIP. https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/Introduction_to_HIP_Programming_Guide.html https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/...
    - Onavo 3y ago
      Didn't Scotus affirm that APIs are not copyrightable?
      - pavon 3y ago
        No, they ruled that APIs are copyrightable, but that Google's re-implementation was fair use. Based on the reasoning of the decision one would expect that in most cases independently reimplementing an API would generally be fair use. However, from a practical point of view, if you are defending yourself in a copyright lawsuit, fair use decisions happen much later in the process and are more subjective. Furthermore, CUDA is a language (dialect of C/C++) not an API, so that precedent may not have much weight.
raphaelj 3y ago
Related question, what is the best way to handle kernel compatibility for CUDA, OpenCL, etc ... ? I had to write a cross-platform kernel a few weeks ago, and I ended using pre-processor guards to make it work with the OpenCL and CUDA compilers [1]. [1] https://github.com/RaphaelJ/libhum/blob/main/libhum/match.kernel https://github.com/RaphaelJ/libhum/blob/main/libhum/match.ke...
- vkaku 3y ago
  Answer is unconventional. Run CI/CD ... It's way easier to see if things will break live when you have it being run-tested on these stacks.
NelsonMinar 3y ago
This looks really interesting but also early days. "this is a very incomplete proof of concept. It's probably not going to work with your application." I hope it develops into something broadly usable!
karim79 3y ago
> Is ZLUDA a drop-in replacement for CUDA? Yes, but certain applications use CUDA in ways which make it incompatible with ZLUDA I think this might get better if and when people redesign their dev workflows, CI/CD pipelines, builds, etc, to deploy code to both hardware platforms to ensure matching functionality and stability. I'm not going to hold my breath just yet. But it would be really great to have two viable platforms/players in this space where code can be run and behave equally.
replete 3y ago
Llama.cpp just added CUDA GPU acceleration yesterday, so this would be very interesting for the emerging space of running local LLMs on commodity hardware. Running CUDA on an AMD RDNA3 APU is what I'd like to see as its probably the cheapest 16GB shared VRAM solution (UMA Frame Buffer BIOS setting) and creates the possibility of running 13b LLM locally on an underutilized iGPU. Aaand its been dead for years, shame.
- brucethemoose2 3y ago
  - llama.cpp already has OpenCL acceleration. It has had it for some time. - AMD already has a CUDA translator: ROCM. It should work with llama.cpp CUDA, but in practice... shrug - Copies the CUDA/OpenCL code make (that are unavoidable for discrete GPUs) are problematic for IGPs. Right now acceleration regresses performance on IGPs. Llama.cpp would need tailor made IGP acceleration. And I'm not even sure what API has the most appropriate zero copy mechanism. Vulkan? OneAPI? Something inside ROCM?
  - sroussey 3y ago
    Apple has a way for zero copy since they have one memory pool for both GPU and cpu. But… I don’t know if it’s possible to do for iGPUs that partition memory in BIOS. I am curious for the answer.
trostaft 3y ago
This project has been unmaintained for a while. Both Intel and AMD have the opportunity to create some actual competition for NVIDIA in the GPGPU space. Intel, at least, I can forgive since they only just entered the market. Why AMD has struggled so hard to get anything going for so long, I don't know...
- easythrees 3y ago
  AMD has HIP.
  - trostaft 3y ago
    Yes, but I don't think its debatable to say that the entire ecosystem is firmly behind NVIDIA's. Usually it comes as a surprise when something does support their framework, whether directly ROCm or even HIP which should be easier.... I shouldn't be surprised that AMD's ecosystem is lagging behind, since their GPU division spent a good decade suffering to be even relevant. Not to mention that NVIDIA has spent a lot of effort on their HPC tools. I don't want this to be too negative towards AMD, they have been steadily growing in this space. Some things do work well, e.g. stable diffusion is totally fine on AMD GPUs. So they seem to be catching up. I just feel a little impatient, especially since their cards are more than powerful enough to be useful. I suppose my point is that the gap in HPC software between NVIDIA and AMD is much larger than the actual capability gap in their hardware, and that's a shame.
    - Eisenstein 3y ago
      Apple was a few months from bankruptcy during most of the 90s competing with IBM and Microsoft, then turned around to become the most profitable company on the planet. It takes a leader and a plan and a lot of talent and the exact right conditions, but industry behemoths get pulled down from the top spot all the time.
      - tracker1 3y ago
        Apple's success is mostly UX and marketing with a walled garden for an application tax. AMD has to actually achieve on the hardware side, not just marketing. Beyond this, AMD has demonstrated that they are, indeed working on closing and pulling ahead. AMD is well ahead of Intel on the server CPU front, they're neck and neck on desktop, with spans ahead in the past few years. And on the GPU side, they've closed a lot of gaps. While I am a little bit of a fan of AMD, there's still work to do. I think AMD really needs to take advantage of their production margins to gain more market share. They also need to get something a bit closer to the 4090 on a performance gpu + entry workstation api/gpgpu workload card. The 7900 XTX is really close, but if they had something with say 32-48gb vram in the sub-2000 space it would really get a lot of the hobbiest and soho types to consider them.
rapatel0 3y ago
Intel should pour money into this project until the code is hosted in scrooge's money bin
SomeRndName11 3y ago
[flagged]
jlebar 3y ago
Oh wow, I think this translates PTX (nvidia's high-level assembly code) to SPIR-V? Am I reading this right? That's...a lot. A note to any systems hackers who might try something like this, you can also retarget clang's CUDA support to SPIR-V via a LLVM-to-SPIR-V translator. I can say with confidence that this works. :)
misterbishop 3y ago
Why is this trending when there hasn't been a commit since Jan 2021? There's a comment here like "it's early days"... the repo has been dormant for longer than it was active.
zackmorris 3y ago
Are there any projects that go the opposite direction, to run CPU code on GPU? I understand that there might be limitations, like not being able to access system calls or the filesystem. What I'm mainly looking for is a way to write C-style code and have it run auto-parallelized, without having to drop into a different language or annotate my code or manually manage buffers.
- andy_ppp 3y ago
  I think C code is full of branches and CPUs are designed to guess and parallelise those (possible) decisions. Graphics cards are designed for running tiny programs against millions of pixels per second. I’m not sure it possible to make these two different concepts the same.
Tade0 3y ago
I appreciate how the name translates to "delusion", considering how "cuda" translates to "miracles" in the same language (Polish).
raphlinus 3y ago
This is something that fundamentally can't work, unfortunately. One showstopper (and there may be others) is subgroup size. Nvidia hardware has a subgroup (warp) size of 32, while Intel's subgroup size story is far more complicated, and depends on a compiler heuristic to tune. The short version of the story is that it's usually 16 but can be 8 if there's a lot of register pressure, or 32 for a big workgroup and not much register pressure (and for those who might reasonably question whether forcing subgroup size to 32 can solve the compatibility issue, the answer is that it will frequently cause registers to spill and performance to tank). CUDA code is not written to be agile in subgroup size, so there is no automated translation that works efficiently on Intel GPU hardware. Longer term, I think we can write GPU code that is portable, but it will require building out the infrastructure for it. Vulkan compute shaders are one good starting point, and as of Vulkan 1.3 the "subgroup size control" feature is mandatory. WebGPU is another possible path to get there, but it's currently lacking a lot of important features, including subgroups at all. There's more discussion of subgroups as a potential WebGPU feature in [1], including how to handle subgroup size. [1]: https://github.com/gpuweb/gpuweb/issues/3950 https://github.com/gpuweb/gpuweb/issues/3950
- AnthonyMouse 3y ago
  Things like this are often useful even if they're not optimal. Before you had a piece of code that simply would not run on your GPU. Now it runs. Even if it's slower than it should be, that's better than not running at all. Which makes more people willing to buy the GPU. Then they go to the developers and ask why the implementation isn't optimized for this hardware lots of people have and the solution is to do an implementation in Vulkan etc.
- pjmlp 3y ago
  Only if SPIR-V tooling ever gets half as good as PTX ecosystem.
  - deleted 3y ago
    [deleted]
- varelse 3y ago
tgtweak 3y ago
How good is the tool at reporting missing cuda functionally?
pjmlp 3y ago
CUDA is a polyglot development enviroment, usually all these projects fail short to focus only on C++. I failed to find information regarding Fortran, Haskell, Julia, .NET, Java support for CUDA workloads.
dang 3y ago
Related: Zluda: CUDA on Intel GPUs - https://news.ycombinator.com/item?id=26262038 https://news.ycombinator.com/item?id=26262038 - Feb 2021 (77 comments)
vkaku 3y ago
Beautiful! Waiting for someone to use this and get benchmarks with PyTorch now.
varelse 3y ago
[dead]

237 points by goranmoomin 3y ago

> Is ZLUDA a drop-in replacement for CUDA? Yes, but certain applications use CUDA in ways which make it incompatible with ZLUDA So no then.

jjallen 3y ago
yeah would probably be good to include where it doesn't work. 1% of the time? 10%?
- mattkrause 3y ago
  It's near the bottom: "What is the status of the project? This project is a Proof of Concept. About the only thing that works currently is Geekbench. It's amazingly buggy and incomplete. You should not rely on it for anything serious."
meepmorp 3y ago
50% of the time it works every time.
- catchnear4321 3y ago
  that’s not quite accurate. it works 100% of the time. until it does not.
glimshe 3y ago
It's a drop-in replacement in the sense that you don't need to modify your code. But it has limitations/incompatibilities. Contrast to something that isn't a drop-in replacement... That would require changes to the application.
- 01100011 3y ago
  This statement makes no sense. "It's compatible with CUDA as long as you don't use all the features of CUDA." So it's a drop in replacement for some subset of modern CUDA. I feel like most folks who are upvoting this don't program CUDA professionally or aren't very advanced in their usage of it.
  - dragonwriter 3y ago
    “Drop-in” = to the extent it works, it requires no app changes. “Complete” = it covers everything. It is drop-in, but not complete.
    - dahart 3y ago
      The issue being that in the original question “drop-in” implies complete, that’s what being a drop-in replacement actually means in other contexts. If it’s not complete, then it’s not really drop-in, even though I don’t necessarily disagree with your definition. You can be right, and parent can be right too, IMHO. The FAQ question is stated ambiguously and in misleadingly black and white terms, and the answer really does look kinda funny starting with the word “Yes” and then following that with “but… not exactly”. Wouldn’t it be better to say drop-in is the goal, and because it’s not complete, we’re not there yet?
  - glimshe 3y ago
    Does the statement make no sense or you simply don't like it? I can understand what it says, sounds like plain English to me. I think that what you're trying to say is that before claiming to be a "drop-in replacement", make sure that your supported feature set is representative enough of mainline CUDA development.

YetAnotherNick 3y ago

If a single dev could do it, why can't AMD do the same for their GPUs.

simcop2387 3y ago
Patents and software licenses probably.
mrstone 3y ago
I imagine they could, but it is probably more of a legal thing.
- viewtransform 3y ago
  Yes it is a legal issue. AMD cannot implement CUDA. However, they have gone around that by creating HIP which is a CUDA adjacent language that runs on AMD and also translates to CUDA for Nvidia GPUs. There is also the HIPify tool to automatically convert existing sources from CUDA to HIP. https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/Introduction_to_HIP_Programming_Guide.html https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/...
  - Onavo 3y ago
    Didn't Scotus affirm that APIs are not copyrightable?
    - pavon 3y ago
      No, they ruled that APIs are copyrightable, but that Google's re-implementation was fair use. Based on the reasoning of the decision one would expect that in most cases independently reimplementing an API would generally be fair use. However, from a practical point of view, if you are defending yourself in a copyright lawsuit, fair use decisions happen much later in the process and are more subjective. Furthermore, CUDA is a language (dialect of C/C++) not an API, so that precedent may not have much weight.

raphaelj 3y ago

Related question, what is the best way to handle kernel compatibility for CUDA, OpenCL, etc ... ? I had to write a cross-platform kernel a few weeks ago, and I ended using pre-processor guards to make it work with the OpenCL and CUDA compilers [1]. [1] https://github.com/RaphaelJ/libhum/blob/main/libhum/match.kernel https://github.com/RaphaelJ/libhum/blob/main/libhum/match.ke...

vkaku 3y ago
Answer is unconventional. Run CI/CD ... It's way easier to see if things will break live when you have it being run-tested on these stacks.

NelsonMinar 3y ago

This looks really interesting but also early days. "this is a very incomplete proof of concept. It's probably not going to work with your application." I hope it develops into something broadly usable!

karim79 3y ago

> Is ZLUDA a drop-in replacement for CUDA? Yes, but certain applications use CUDA in ways which make it incompatible with ZLUDA I think this might get better if and when people redesign their dev workflows, CI/CD pipelines, builds, etc, to deploy code to both hardware platforms to ensure matching functionality and stability. I'm not going to hold my breath just yet. But it would be really great to have two viable platforms/players in this space where code can be run and behave equally.

replete 3y ago

Llama.cpp just added CUDA GPU acceleration yesterday, so this would be very interesting for the emerging space of running local LLMs on commodity hardware. Running CUDA on an AMD RDNA3 APU is what I'd like to see as its probably the cheapest 16GB shared VRAM solution (UMA Frame Buffer BIOS setting) and creates the possibility of running 13b LLM locally on an underutilized iGPU. Aaand its been dead for years, shame.

brucethemoose2 3y ago
- llama.cpp already has OpenCL acceleration. It has had it for some time. - AMD already has a CUDA translator: ROCM. It should work with llama.cpp CUDA, but in practice... shrug - Copies the CUDA/OpenCL code make (that are unavoidable for discrete GPUs) are problematic for IGPs. Right now acceleration regresses performance on IGPs. Llama.cpp would need tailor made IGP acceleration. And I'm not even sure what API has the most appropriate zero copy mechanism. Vulkan? OneAPI? Something inside ROCM?
- sroussey 3y ago
  Apple has a way for zero copy since they have one memory pool for both GPU and cpu. But… I don’t know if it’s possible to do for iGPUs that partition memory in BIOS. I am curious for the answer.

trostaft 3y ago

This project has been unmaintained for a while. Both Intel and AMD have the opportunity to create some actual competition for NVIDIA in the GPGPU space. Intel, at least, I can forgive since they only just entered the market. Why AMD has struggled so hard to get anything going for so long, I don't know...

easythrees 3y ago
AMD has HIP.
- trostaft 3y ago
  Yes, but I don't think its debatable to say that the entire ecosystem is firmly behind NVIDIA's. Usually it comes as a surprise when something does support their framework, whether directly ROCm or even HIP which should be easier.... I shouldn't be surprised that AMD's ecosystem is lagging behind, since their GPU division spent a good decade suffering to be even relevant. Not to mention that NVIDIA has spent a lot of effort on their HPC tools. I don't want this to be too negative towards AMD, they have been steadily growing in this space. Some things do work well, e.g. stable diffusion is totally fine on AMD GPUs. So they seem to be catching up. I just feel a little impatient, especially since their cards are more than powerful enough to be useful. I suppose my point is that the gap in HPC software between NVIDIA and AMD is much larger than the actual capability gap in their hardware, and that's a shame.
  - Eisenstein 3y ago
    Apple was a few months from bankruptcy during most of the 90s competing with IBM and Microsoft, then turned around to become the most profitable company on the planet. It takes a leader and a plan and a lot of talent and the exact right conditions, but industry behemoths get pulled down from the top spot all the time.
    - tracker1 3y ago
      Apple's success is mostly UX and marketing with a walled garden for an application tax. AMD has to actually achieve on the hardware side, not just marketing. Beyond this, AMD has demonstrated that they are, indeed working on closing and pulling ahead. AMD is well ahead of Intel on the server CPU front, they're neck and neck on desktop, with spans ahead in the past few years. And on the GPU side, they've closed a lot of gaps. While I am a little bit of a fan of AMD, there's still work to do. I think AMD really needs to take advantage of their production margins to gain more market share. They also need to get something a bit closer to the 4090 on a performance gpu + entry workstation api/gpgpu workload card. The 7900 XTX is really close, but if they had something with say 32-48gb vram in the sub-2000 space it would really get a lot of the hobbiest and soho types to consider them.

rapatel0 3y ago

Intel should pour money into this project until the code is hosted in scrooge's money bin

SomeRndName11 3y ago

[flagged]

jlebar 3y ago

Oh wow, I think this translates PTX (nvidia's high-level assembly code) to SPIR-V? Am I reading this right? That's...a lot. A note to any systems hackers who might try something like this, you can also retarget clang's CUDA support to SPIR-V via a LLVM-to-SPIR-V translator. I can say with confidence that this works. :)

misterbishop 3y ago

Why is this trending when there hasn't been a commit since Jan 2021? There's a comment here like "it's early days"... the repo has been dormant for longer than it was active.

zackmorris 3y ago

Are there any projects that go the opposite direction, to run CPU code on GPU? I understand that there might be limitations, like not being able to access system calls or the filesystem. What I'm mainly looking for is a way to write C-style code and have it run auto-parallelized, without having to drop into a different language or annotate my code or manually manage buffers.

andy_ppp 3y ago
I think C code is full of branches and CPUs are designed to guess and parallelise those (possible) decisions. Graphics cards are designed for running tiny programs against millions of pixels per second. I’m not sure it possible to make these two different concepts the same.

Tade0 3y ago

I appreciate how the name translates to "delusion", considering how "cuda" translates to "miracles" in the same language (Polish).

raphlinus 3y ago

This is something that fundamentally can't work, unfortunately. One showstopper (and there may be others) is subgroup size. Nvidia hardware has a subgroup (warp) size of 32, while Intel's subgroup size story is far more complicated, and depends on a compiler heuristic to tune. The short version of the story is that it's usually 16 but can be 8 if there's a lot of register pressure, or 32 for a big workgroup and not much register pressure (and for those who might reasonably question whether forcing subgroup size to 32 can solve the compatibility issue, the answer is that it will frequently cause registers to spill and performance to tank). CUDA code is not written to be agile in subgroup size, so there is no automated translation that works efficiently on Intel GPU hardware. Longer term, I think we can write GPU code that is portable, but it will require building out the infrastructure for it. Vulkan compute shaders are one good starting point, and as of Vulkan 1.3 the "subgroup size control" feature is mandatory. WebGPU is another possible path to get there, but it's currently lacking a lot of important features, including subgroups at all. There's more discussion of subgroups as a potential WebGPU feature in [1], including how to handle subgroup size. [1]: https://github.com/gpuweb/gpuweb/issues/3950 https://github.com/gpuweb/gpuweb/issues/3950

AnthonyMouse 3y ago
Things like this are often useful even if they're not optimal. Before you had a piece of code that simply would not run on your GPU. Now it runs. Even if it's slower than it should be, that's better than not running at all. Which makes more people willing to buy the GPU. Then they go to the developers and ask why the implementation isn't optimized for this hardware lots of people have and the solution is to do an implementation in Vulkan etc.
pjmlp 3y ago
Only if SPIR-V tooling ever gets half as good as PTX ecosystem.
- deleted 3y ago
  [deleted]
varelse 3y ago

tgtweak 3y ago

How good is the tool at reporting missing cuda functionally?

pjmlp 3y ago

CUDA is a polyglot development enviroment, usually all these projects fail short to focus only on C++. I failed to find information regarding Fortran, Haskell, Julia, .NET, Java support for CUDA workloads.

dang 3y ago

Related: Zluda: CUDA on Intel GPUs - https://news.ycombinator.com/item?id=26262038 https://news.ycombinator.com/item?id=26262038 - Feb 2021 (77 comments)

vkaku 3y ago

Beautiful! Waiting for someone to use this and get benchmarks with PyTorch now.

varelse 3y ago

[dead]