4 ms·

ILGPU: Write GPU programs with C# and F#

157 points by neonsunset 2y ago

jvanderbot 2y ago
I'm going through the CUDA courses. I've done GPU and CPU optimization as an enthusiastic amateur in my day job once or twice a year, but it's not my core focus. It quickly seems that the low level C/Cpp is becoming obsolete, and it's hard to squeeze performance unless you're doing something truly green field / new. Otherwise someone has already optimized the hell out of it. So what's the use case for porting GPU to higher level languages like C#? What would you use this for?
- gtani 2y ago
  what do you mean obsolete, read about PTX target tests: https://old.reddit.com/r/rust/comments/18209in/gpu_programming_in_rust/kagca1r/ https://old.reddit.com/r/rust/comments/18209in/gpu_programmi...
- sheepscreek 2y ago
  Make it more accessible? Prototyping? Eg. easily determining if your use case is even suited for a GPU workload. Best case - it is, and now you can write a custom CUDA kernel to squeeze out more performance. Worst case, you lose a couple of hours vs weeks before you discover it’s not going to work.
- neonsunset 2y ago
  Given IL itself is an abstract stack-based bytecode, it can be compiled to the corresponding IR, which can then target corresponding back-end (CUDA, OpenCL, CPU, etc.) - this is what ILGPU does. Because all code is in the single repository in the post and is fairly easy to read, you can skim through it to draw your own conclusions if this interests you. Also, very easy to start using: just `dotnet add package ILGPU` on most configurations (as ADHD puts higher mental strain on activities involving complex configuration, I try to keep to the tools that have minimal ceremony) C# (and F# by extension) generally allow to write system-ish code, with references to locals and same C primitives, which means that you're likely not sacrificing in performance in this particular scenario by having the language be higher-level. After all, you're using ILGPU's APIs first and foremost. As to why use it at all - you are likely to move faster with it than C++, especially if it's not your full-time job, with all the escape hatches to extract 99.9% efficiency still on the table (that is, if performance of the kernel emitted by ILGPU has issues in the first place - see below for alternative, cheap FFI and easy C/C++ integration are still there as well). It also lets you do things like PTX assembly: https://github.com/m4rs-mt/ILGPU/blob/master/Samples/InlinePTXAssembly/Program.cs https://github.com/m4rs-mt/ILGPU/blob/master/Samples/InlineP...
- almostgotcaught 2y ago
  > It quickly seems that the low level C/Cpp is becoming obsolete, and it's hard to squeeze performance unless you're doing something truly green field / new. Otherwise someone has already optimized the hell out of it. I hear and see both of these sentiments frequently from the internet crowd (on-lookers). It's both wrong and humorously arrogant. I'll repeat what I said to someone on reddit yesterday: there are thousands (maybe 10s?) scattered around the FAANGs, NVIDIA, AMD, Intel, accelerator startups, boutiques, etc. whose day to day is both C/C++ and squeezing perf out of kernels and getting many points (sometimes 10s) improvement. Certainly the wins aren't daily but I'm saying they do steadily find room for improvement. How is that possible? I'll give you a hint: platforms, demands, hardware, use-cases all change essentially on a quarterly basis. So before you proclaim victory on behalf of whatever high-level framework, ask yourself if you're really familiar with the production and business environment for this kind of code.
  - jvanderbot 2y ago
    As a learner and self proclaimed amateur, I assume the vitriolic tone is directed elsewhere. Without that, you're saying that performant pipelines are definitely in demand and optimization is still a full time job. That's good! I'd like to get better at that.
    - almostgotcaught 2y ago
      >I assume the vitriolic tone is directed elsewhere I'm always really shocked on hn when people getting called out for arrogance respond with accusations (of vitriol). What day job do you have where you can make proclamations like "It quickly seems that the low level C/Cpp is becoming obsolete" while simultaneously admitting being an amateur and not get checked. Must be very different from my day job where being precise and accurate and conservative is paramount. Moreover what kind of habit of thought are you in that being called out for that is read as vitriol?
      - ClassyJacket 2y ago
        Oh my god, calm down. Jesus christ.
- deleted 2y ago
  [deleted]
- jay-barronville 2y ago
  1. C and C++ aren’t going anywhere. 2. I don’t think you have an appreciation for the sheer amount of software—including lots of very critical software—implemented in C and C++ that are being improved upon daily. A buddy of mine works on critical low-level software (including C) in the energy sector. Millions of lines of code. The effects, positive or negative, of any single code change can impact tens of thousands of Americans (likely even more). His job is to maintain and continuously optimize this software. Given your comment, I think you’d be surprised to learn that he never runs out of work.
  - jvanderbot 2y ago
    Well, I write in those languages every day! It's not a statement about the languages, just wondering about whether CUDA is best approached low level or from well established libraries.
  - vmsp 2y ago
    Would you mind expanding on what your friend does? Sounds like energy trading which is something I’ve been taking a look at. I’m just curious.
    - AndrewDucker 2y ago
      Seems more likely it's to do with running the systems and networks.
- deleted 2y ago
  [deleted]
- fulafel 2y ago
  Programmers generally use higher level languages to implement original code because the code is easier for humans to read, write and debug. Porting from low level languages (eg CUDA or C++ to F#) is less common.
- nullandkale 2y ago
  I use ILGPU all the time for prototyping different image processing and rendering things for work and for my personal projects. I use cpp for a few things where the extra efficiency is worth it, but for most tasks c# is plenty fast enough.
davidgl 2y ago
Also see https://www.fshade.org/ https://www.fshade.org/, a F# dsl for shaders
atonalfreerider 2y ago
Also recommending ComputeSharp. I've been using it for a few years now: https://github.com/Sergio0694/ComputeSharp https://github.com/Sergio0694/ComputeSharp
- neonsunset 2y ago
  Unfortunately, ComputeSharp is Windows-only which significantly limits its usability.
nullandkale 2y ago
I use ILGPU all the time! You would be amazed the performance you can get out of c# if you are careful about how you write your code. I recently wrote a super fast gaussian splat renderer in c# and used ILGPU to sort the splats on the gpu. You can even do weird cuda stuff like pass buffers between opengl and cuda, all in c#! https://github.com/NullandKale/OpenTKSplat/blob/main/OpenTKSplat/Compute/ILGPUOpenGLExchangeBuffer.cs https://github.com/NullandKale/OpenTKSplat/blob/main/OpenTKS...
ein0p 2y ago
This needs a Metal backend IMO. Higher level GPU programming on a Mac is a bit of a shitshow, and Apple devices do have pretty potent GPUs. They’re also largely compatible across their entire product line, thus dramatically expanding the potential installed base of the output of such toolkits, assuming it can be AOT compiled.
- nullandkale 2y ago
  Currently ILGPU already supports arm64 linux if you have a cuda or opencl device, so supporting apple silicon is not out of the question. It has been floated that we would support Metal via Vulkan support and MoltenVK but not much work has been done on that front. Currently SIMD support for a fast CPU accelerator is the main focus for the next big accelerator type.
  - ein0p 2y ago
    Note also that Apple SoCs also have a matrix unit, called AMX, which is designed specifically for matmuls. You could expand the math throughput of the CPU quite dramatically by taking advantage of it. I don’t think there’s any official public documentation on it, but here’s a GitHub repo that documents it independently https://github.com/corsix/amx https://github.com/corsix/amx
    - nullandkale 2y ago
      We have thought of supporting the tensor cores in cuda devices as well, we could probably use the same abstractions we need for that for the amx support. Unfortunately we mainly focus on cuda support because most cases people are using cuda for gpu compute purposes.
      - ein0p 2y ago
        CUDA is a bit of a well-trodden ground, you aren’t going to do much better there (if at all) than cuBLAS and cuDNN. But I get what you’re saying, gotta pick one’s battles.
DeathArrow 2y ago
Not exactly the same thing, but there is a very nice ML framework for .Net called ML.Net.

157 points by neonsunset 2y ago

I'm going through the CUDA courses. I've done GPU and CPU optimization as an enthusiastic amateur in my day job once or twice a year, but it's not my core focus. It quickly seems that the low level C/Cpp is becoming obsolete, and it's hard to squeeze performance unless you're doing something truly green field / new. Otherwise someone has already optimized the hell out of it. So what's the use case for porting GPU to higher level languages like C#? What would you use this for?

gtani 2y ago
what do you mean obsolete, read about PTX target tests: https://old.reddit.com/r/rust/comments/18209in/gpu_programming_in_rust/kagca1r/ https://old.reddit.com/r/rust/comments/18209in/gpu_programmi...
sheepscreek 2y ago
Make it more accessible? Prototyping? Eg. easily determining if your use case is even suited for a GPU workload. Best case - it is, and now you can write a custom CUDA kernel to squeeze out more performance. Worst case, you lose a couple of hours vs weeks before you discover it’s not going to work.
neonsunset 2y ago
Given IL itself is an abstract stack-based bytecode, it can be compiled to the corresponding IR, which can then target corresponding back-end (CUDA, OpenCL, CPU, etc.) - this is what ILGPU does. Because all code is in the single repository in the post and is fairly easy to read, you can skim through it to draw your own conclusions if this interests you. Also, very easy to start using: just `dotnet add package ILGPU` on most configurations (as ADHD puts higher mental strain on activities involving complex configuration, I try to keep to the tools that have minimal ceremony) C# (and F# by extension) generally allow to write system-ish code, with references to locals and same C primitives, which means that you're likely not sacrificing in performance in this particular scenario by having the language be higher-level. After all, you're using ILGPU's APIs first and foremost. As to why use it at all - you are likely to move faster with it than C++, especially if it's not your full-time job, with all the escape hatches to extract 99.9% efficiency still on the table (that is, if performance of the kernel emitted by ILGPU has issues in the first place - see below for alternative, cheap FFI and easy C/C++ integration are still there as well). It also lets you do things like PTX assembly: https://github.com/m4rs-mt/ILGPU/blob/master/Samples/InlinePTXAssembly/Program.cs https://github.com/m4rs-mt/ILGPU/blob/master/Samples/InlineP...
almostgotcaught 2y ago
> It quickly seems that the low level C/Cpp is becoming obsolete, and it's hard to squeeze performance unless you're doing something truly green field / new. Otherwise someone has already optimized the hell out of it. I hear and see both of these sentiments frequently from the internet crowd (on-lookers). It's both wrong and humorously arrogant. I'll repeat what I said to someone on reddit yesterday: there are thousands (maybe 10s?) scattered around the FAANGs, NVIDIA, AMD, Intel, accelerator startups, boutiques, etc. whose day to day is both C/C++ and squeezing perf out of kernels and getting many points (sometimes 10s) improvement. Certainly the wins aren't daily but I'm saying they do steadily find room for improvement. How is that possible? I'll give you a hint: platforms, demands, hardware, use-cases all change essentially on a quarterly basis. So before you proclaim victory on behalf of whatever high-level framework, ask yourself if you're really familiar with the production and business environment for this kind of code.
- jvanderbot 2y ago
  As a learner and self proclaimed amateur, I assume the vitriolic tone is directed elsewhere. Without that, you're saying that performant pipelines are definitely in demand and optimization is still a full time job. That's good! I'd like to get better at that.
  - almostgotcaught 2y ago
    >I assume the vitriolic tone is directed elsewhere I'm always really shocked on hn when people getting called out for arrogance respond with accusations (of vitriol). What day job do you have where you can make proclamations like "It quickly seems that the low level C/Cpp is becoming obsolete" while simultaneously admitting being an amateur and not get checked. Must be very different from my day job where being precise and accurate and conservative is paramount. Moreover what kind of habit of thought are you in that being called out for that is read as vitriol?
    - ClassyJacket 2y ago
      Oh my god, calm down. Jesus christ.
deleted 2y ago
[deleted]
jay-barronville 2y ago
1. C and C++ aren’t going anywhere. 2. I don’t think you have an appreciation for the sheer amount of software—including lots of very critical software—implemented in C and C++ that are being improved upon daily. A buddy of mine works on critical low-level software (including C) in the energy sector. Millions of lines of code. The effects, positive or negative, of any single code change can impact tens of thousands of Americans (likely even more). His job is to maintain and continuously optimize this software. Given your comment, I think you’d be surprised to learn that he never runs out of work.
- jvanderbot 2y ago
  Well, I write in those languages every day! It's not a statement about the languages, just wondering about whether CUDA is best approached low level or from well established libraries.
- vmsp 2y ago
  Would you mind expanding on what your friend does? Sounds like energy trading which is something I’ve been taking a look at. I’m just curious.
  - AndrewDucker 2y ago
    Seems more likely it's to do with running the systems and networks.
deleted 2y ago
[deleted]
fulafel 2y ago
Programmers generally use higher level languages to implement original code because the code is easier for humans to read, write and debug. Porting from low level languages (eg CUDA or C++ to F#) is less common.
nullandkale 2y ago
I use ILGPU all the time for prototyping different image processing and rendering things for work and for my personal projects. I use cpp for a few things where the extra efficiency is worth it, but for most tasks c# is plenty fast enough.

davidgl 2y ago

Also see https://www.fshade.org/ https://www.fshade.org/, a F# dsl for shaders

atonalfreerider 2y ago

Also recommending ComputeSharp. I've been using it for a few years now: https://github.com/Sergio0694/ComputeSharp https://github.com/Sergio0694/ComputeSharp

neonsunset 2y ago
Unfortunately, ComputeSharp is Windows-only which significantly limits its usability.

nullandkale 2y ago

I use ILGPU all the time! You would be amazed the performance you can get out of c# if you are careful about how you write your code. I recently wrote a super fast gaussian splat renderer in c# and used ILGPU to sort the splats on the gpu. You can even do weird cuda stuff like pass buffers between opengl and cuda, all in c#! https://github.com/NullandKale/OpenTKSplat/blob/main/OpenTKSplat/Compute/ILGPUOpenGLExchangeBuffer.cs https://github.com/NullandKale/OpenTKSplat/blob/main/OpenTKS...

ein0p 2y ago

This needs a Metal backend IMO. Higher level GPU programming on a Mac is a bit of a shitshow, and Apple devices do have pretty potent GPUs. They’re also largely compatible across their entire product line, thus dramatically expanding the potential installed base of the output of such toolkits, assuming it can be AOT compiled.

nullandkale 2y ago
Currently ILGPU already supports arm64 linux if you have a cuda or opencl device, so supporting apple silicon is not out of the question. It has been floated that we would support Metal via Vulkan support and MoltenVK but not much work has been done on that front. Currently SIMD support for a fast CPU accelerator is the main focus for the next big accelerator type.
- ein0p 2y ago
  Note also that Apple SoCs also have a matrix unit, called AMX, which is designed specifically for matmuls. You could expand the math throughput of the CPU quite dramatically by taking advantage of it. I don’t think there’s any official public documentation on it, but here’s a GitHub repo that documents it independently https://github.com/corsix/amx https://github.com/corsix/amx
  - nullandkale 2y ago
    We have thought of supporting the tensor cores in cuda devices as well, we could probably use the same abstractions we need for that for the amx support. Unfortunately we mainly focus on cuda support because most cases people are using cuda for gpu compute purposes.
    - ein0p 2y ago
      CUDA is a bit of a well-trodden ground, you aren’t going to do much better there (if at all) than cuBLAS and cuDNN. But I get what you’re saying, gotta pick one’s battles.

DeathArrow 2y ago

Not exactly the same thing, but there is a very nice ML framework for .Net called ML.Net.