8 ms·

Using Libc for GPUs

193 points by hochmartinez 2y ago

almostgotcaught 2y ago
this was presented at llvm this year https://www.youtube.com/watch?v=4TxGWis1mws https://www.youtube.com/watch?v=4TxGWis1mws - it was a nice talk.
kelsey98765431 2y ago
Running normal c directly on gpu has been the dream for a long time. this looks excellent
- C-programmer 2y ago
  Genuinely, why? - For new code, all of the functions [here](https://libc.llvm.org/gpu/support.html#libc-gpu-support https://libc.llvm.org/gpu/support.html#libc-gpu-support) you can do without just fine. - For old code: * Your project is large enough that you are likely use using an unsupported libc function somewhere. * Your project is small enough that you would benefit from just implementing a new kernel yourself. I am biased because I avoid the C standard library even on the CPU, but this seems like a technology that raises the floor not the ceiling of what is possible.
  - tredre3 2y ago
    > this seems like a technology that raises the floor not the ceiling of what is possible. In your view, how is making GPU programming easier a bad thing?
    - convolvatron 2y ago
      that's clearly not a bad thing. however encouraging people to run mutating, procedural code with explicit loops and aliasing maybe isn't the right path to get there. particularly if you just drag forward all the weird old baggage with libc and its horrible string conventions. I think any programming environment that treats a gpu as a really slow serial cpu isn't really what you want(?)
      - quotemstr 2y ago
        What if it encourages people to write parallel and functional code on CPUs? That'd be a good thing. Influence works both ways. The bigger problem is that GPUs have various platform features (shared memory, explicit cache residency and invalidation management) that CPUs sadly don't yet. Sure, you could expose these facilities via compiler intrinsics, but then you end up code that might be syntactically valid C but is alien both to CPUs and human minds
        fc417fc802 2y ago
        > is alien both to CPUs and human minds On the contrary I would love that. The best case scenario in my mind is being able to express the native paradigms of all relevant platforms while writing a single piece of code that can then be compiled for any number of backends and dynamically retargeted between them at runtime. It would make debugging and just about everything else SO MUCH EASIER. The equivalent of being able to compile some subset of functions for both ARM and x86 and then being able to dynamically dispatch to either version at runtime, except replace ARM with a list of all the GPU ISAs that you care about.
  - JonChesterfield 2y ago
    One thing this gives you is syscall on the gpu. Functions like sprintf are just blobs of userspace code, but others like fopen require support from the operating system (or whatever else the hardware needs you to do). That plumbing was decently annoying to write for the gpu. These aren't gpu kernels. They're functions to call from kernels.
    - almostgotcaught 2y ago
      > One thing this gives you is syscall on the gpu i wish people in our industry would stop (forever, completely, absolutely) using metaphors/allusions. it's a complete disservice to anyone that isn't in on the trick. it doesn't give you syscalls. that's impossible because there's no sys/os on a gpu and your actual os does not (necessarily) have any way to peer into the address space/schedular/etc of a gpu core. what it gives you is something that's working really really hard to pretend be a syscall: > Traditionally, the C library abstracts over several functions that interface with the platform’s operating system through system calls. The GPU however does not provide an operating system that can handle target dependent operations. Instead, we implemented remote procedure calls to interface with the host’s operating system while executing on a GPU. https://libc.llvm.org/gpu/rpc.html https://libc.llvm.org/gpu/rpc.html.
      - quotemstr 2y ago
        It's a matter of perspective. If you think of the GPU as a separate computer, you're right. If you think of it as a coprocessor, then the use of RPC is just an implementation detail of the system call mechanism, not a semantically different thing. When an old school 486SX delegates a floating point instruction to a physically separate 487DX coprocessor, is it executing an instruction or doing an RPC? If RPC, does the same instruction start being a real instruction when you replace your 486SX with a 486DX, with an integrated GPU? The program can't tell the difference!
        almostgotcaught 2y ago
        > It's a matter of perspective. If you think of the GPU as a separate computer, you're right. this perspective is a function of exactly one thing: do you care about the performance of your program? if not then sure indulge in whatever abstract perspective you want ("it's magic, i just press buttons and the lights blink"). but if you don't care about perf then why are you using a GPU at all...? so for people that aren't just randomly running code on a GPU (for shits and giggles), the distinction is very significant between "syscall" and syscall. people who say these things don't program GPUs for a living. there are no abstractions unless you don't care about your program's performance (in which case why are you using a GPU at all).
  - nickysielicki 2y ago
    https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/#unified_memory_after_hmm https://developer.nvidia.com/blog/simplifying-gpu-applicatio...
  - deleted 2y ago
    [deleted]
  - JonChesterfield 2y ago
    > Genuinely, why? > ... this seems like a technology that raises the floor not the ceiling of what is possible. The root cause reason for this project existing is to show that GPU programming is not synonymous with CUDA (or the other offloading languages). It's nominally to help people run existing code on GPUs. Disregarding that use case, it shows that GPUs can actually do things like fprintf or open sockets. This is obvious to the implementation but seems largely missed by application developers. Lots of people think GPUs can only do floating point math. Especially on an APU, where the GPU units and the CPU cores can hammer on the same memory, it is a travesty to persist with the "offloading to accelerator" model. Raw C++ isn't an especially sensible language to program GPUs in but it's workable and I think it's better than CUDA.
    - rbanffy 2y ago
      > Lots of people think GPUs can only do floating point math. IIRC, every Raspberry Pi is brought up by the GPU setting up the system before the CPU is brought out of reset and the bootloader looks for the OS. > it is a travesty to persist with the "offloading to accelerator" model. Operating systems would need to support heterogeneous processors running programs with different ISAs accessing the same pools of memory. I'd LOVE to see that. It'd be extremely convenient to have first-class processes running on the GPU MIMD cores. I'm not sure there is much research done in that space. I believe IBM mainframe OSs have something like that because programmers are exposed to the various hardware assists that run as coprocessors sharing the main memory with the OS and applications.
      - als0 2y ago
        > I'm not sure there is much research done in that space. There is. And the finest example I can think of is Barrelfish https://barrelfish.org https://barrelfish.org
        rbanffy 2y ago
        Interesting - it resembles a network of heterogeneous systems that can share a memory space used primarily for explicit data exchange. Not quite what I was imagining, but probably much simpler to implement than a Unix where the kernel can see processes running on different ISAs on a shared memory space. I guess hardware availability is an issue, as there aren't many computers with, say, an ARM, a RISC-V, an x86, and an AMD iGPU sharing a common memory pool. OTOH, there are many where a 32-bit ARM shares the memory pool with 64-bit cores. Usually the big cores run applications while the small ARM does housekeeping or other low-latency task.
- amelius 2y ago
  I wonder what C would look like if CPUs would evolve into what GPUs are today. What if a CPU had assembly instructions for everything a GPU can do? Would compiler/language designers support them?
  - gpderetta 2y ago
    there is no reason to wonder: https://ispc.github.io/ https://ispc.github.io/
- quotemstr 2y ago
  I've never understood why people say you "can't" do this or that on GPU. A GPU is made of SMs, and each SM is just a CPU with very wide SIMD pipes and very good hyperthreading. You can take one thread of a warp in a SM and do exactly the same things a CPU would do. Would you get 1/32 potential performance? Sure. But so what? Years ago, we did plenty of useful work with less than 1/32 of a modest CPU, and we can again. One of the more annoying parts of the Nvidia experience is PTX. I. I know perfectly well that your CPU/SM/whatever has a program counter. Let me manipulate it directly!
  - bee_rider 2y ago
    I think people are just saying it isn’t very cost effective to use a whole GPU as 1/32 of a modest (modern?) CPU.
    - quotemstr 2y ago
      Cars are slow and inefficient in reverse gear but a car that couldn't drive in reverse would be broken.
      - bee_rider 2y ago
        You put “can’t” in quotes, so I guess you are quoting somebody, but I don’t see where the quote is from, so I’m not sure what they actually meant. But I suspect they are using “can’t” informally. Like: You can’t run a drag race in reverse. Ok, you technically could, but it would be a silly thing to do.
    - imtringued 2y ago
      Why would that matter? Adding extra CPU cores to the GPU sounds like a much dumber idea to me. You'd be wasting silicon on something that is meant to be used infrequently. Is it really that difficult to imagine 1% of your workload doing some sort of management code, except since it is running on your GPU, you can now freely intersperse those 1% every 50 microseconds without too much overhead?
Archit3ch 2y ago
No direct Metal target.
- almostgotcaught 2y ago
  this is an LLVM project... you want this to work on Metal, ask apple to add a Metal backend to LLVM https://github.com/llvm/llvm-project/tree/main/llvm/lib/Target https://github.com/llvm/llvm-project/tree/main/llvm/lib/Targ...
  - rbanffy 2y ago
    I am surprised there isn't.
- JonChesterfield 2y ago
  No Intel either. The port would be easy - gpuintrin.h abstracts over the intrinsics, provide an implementation for those, write a loader in terms of opencl or whatever if you want to run the test suite. The protocol needs ordered load/store on shared memory but nothing else. I wrote a paper trying to make it clear that load/store on shmem was sufficient which doesn't seem to be considered persuasive. It's specifically designed to tolerate architectures doing slopping things with cache invalidation. It could run much faster with fetch_or / fetch_and instructions (as APUs have, but PCIe does not). It could also hang off DMA but that isn't implemented (I want to have the GPU push packets over the network without involving the x64 CPU at all).
gdiamos 2y ago
Nice to see this finally after 15 years.
amelius 2y ago
I hate libc. It's such a common cause of versioning problems on my system. Can't they just stop making new versions of it?
- wbl 2y ago
  Dynamic linking has some benefits, but many detractors.
  - rbanffy 2y ago
    It's nice not having to recompile all your software because of a CVE impacting libc, or any other fundamental component of the system.
    - adrian_b 2y ago
      In decades of experiencing various computing environments, the amount of time wasted to fix applications broken by updates to dynamically-linked libraries has dwarfed by a few orders of magnitude the amount of time saved by not recompiling statically-linked applications because of a new library version that fixes security problems.
      - rbanffy 2y ago
        What scares me about the need to recompile statically linked binaries is that the problem is invisible until a bug hits. You don't know the statically linked library is vulnerable unless you keep track of all versions that went into that binary and almost no organization does that. DLL problems are very easy to see and very obvious when they happen. But it's been a long while since I last saw one.
  - adrian_b 2y ago
    If some people like dynamically-linked applications, let them have such applications. What annoys me greatly is the existence of unavoidable libraries that have been designed intentionally to not work when linked statically, which make the life difficult for those who prefer to use only statically-linked applications. The main offending library is glibc, but there are too many other such cases.
fuhsnn 2y ago
I wonder how GPU is going to access an unknown size NULL terminated string in system RAM, the strchr() source looks like normal C++. In my minimal Vulkan GPGPU experience the data need to be bound to VkDeviceMemory to be accessible through PCI bus with compute shader, is LLVM libc runtime doing similar set-ups in the background, and if so, is it faster than glibc's hand-tuned AVX implementation?
- JonChesterfield 2y ago
  This is libc running on the GPU, not a libc spanning CPU and GPU. The primary obstruction to doing that is persuading people to let go of glibc. The spec of "host runs antique glibc, GPU runs some other thing that interops transparently with glibc" is a nightmare of hacks and tragedy. What would be relatively easy to put together is llvm libc running on the host x64 and also llvm libc running on the GPU. There's then the option to do things like malloc() on the GPU and free() the same pointer on the CPU. Making it genuinely seamless also involves persuading people to change what function pointers are, do some work on the ABI, and preferably move to APUs because PCIe is unhelpful. There's an uphill battle to bring people along on the journey of "program the things differently". For example, here's a thread trying to drum up enthusiasm for making function pointers into integers as that makes passing them between heterogenous architectures far easier https://discourse.llvm.org/t/rfc-function-pointers-as-integers-beyond-wasm/83208/7 https://discourse.llvm.org/t/rfc-function-pointers-as-intege....
  - fuhsnn 2y ago
    >making function pointers into integers as that makes passing them between heterogenous architectures This is interesting, though function pointers are long expected to be address on binary, C-brained people like me would probably adapt to the concept of "pointer to a heterogeneous lambda object" or "shared id across heterogeneous runtimes" easier.
    - JonChesterfield 2y ago
      Yeah, I might need to take another angle at the branding / sales part, possibly with a prototype in hand. I was hopeful that wasm was sufficient prior art. Whether function pointers are absolute addresses, or relative to the start of text, or resolved through trampolines filled in by the loader, or are offsets into some table is all (nominally) invisible to the language. Integer in [0, k) has the nice feature that multiple GPUs can each index into their own lookup table containing pointers to the local code section. Or for calling into another machine - it's essentially a cheap serialisation / perfect hash done at linker time. It makes indirect calls slower, but indirect calls are slow anyway so hopefully saleable.
einpoklum 2y ago
I am pretty sure this is just a gimmick. I would not call libc code in a GPU kernel. It would mean dragging in a whole bunch of stuff I don't want, and cant control or pick-and-choose. That makes sense for regular processes on the CPU; it does _not_ make sense in code you run millions of times in GPU threads. I see people saying they've "dreamed" of this or have waited so long for this to happen... well, my friends, you should not have; and I'm afraid you're in for a disappointment.
- JonChesterfield 2y ago
  It uses a few mb of contiguous shared memory and periodically calling a function from a host thread. Unless you only want sprintf or similar in which case neither is needed. The unused code deadstrips pretty well. Won't help compilation time. Generally you don't want libc calls in numerical kernels doing useful stuff - the most common request was for printf as a debugging crutch, I mostly wanted mmap.
  - einpoklum 2y ago
    > It uses a few mb of contiguous shared memory and periodically calling a function from a host thread. Well, if someone wants to shoot themselves in the head, then by all means... > Unless you only want sprintf or similar in which case neither is needed. > ... Generally you don't want libc calls in numerical kernels doing useful > stuff - the most common request was for printf as a debugging crutch I have actually adapted a library for that particular case: https://github.com/eyalroz/printf/ https://github.com/eyalroz/printf/ I started with a standalone printf-family implementation targetting embedded devices, and (among other things) adapted it for use also with CUDA. > I mostly wanted mmap. Does it really make sense to make a gazillion mmap calls from the threads of your GPU kernel? I mean, is it really not always better to mmap on the CPU side? At most, I might do it asynchronously using a CUDA callback or some other mechanism. But I will admit I've not had that use-case.
    - rowanG077 2y ago
      There is a ton of code that is sections of massive parallelism interspersed with sections of serial non-hot code. It makes total sense to do some syscall then and continue to the next section.
jhuber6 2y ago
Very cool to see my project posted here! The motivation behind a lot of this was to have community LLVM implementations of runtime functions normally provided by the vendor libraries (C math, printf, malloc), but if you implement one function you may as well implement them all. Realistically, the infrastructure behind this project is more relevant than the C library calls themselves. The examples in the linked documentation can be used for any arbitrary C/C++ just as well as the LLVM C library, it's simply statically linking. This is what allowed me to compile and run more complicated things like libc++ and DOOM on the GPU as well. The RPC interface can also be used to implement custom host services from the GPU, or used to communicate between any two shared memory processes.
- noxa 2y ago
  Just wanted to say thanks for pushing on this front! I'm not using the libc portion but the improvements to clang/llvm that allow this to work have been incredible. When I was looking a few months back the only options that felt practical for writing large amounts of device code were cuda/hip or opencl and a friend suggested I just try C _and it worked_. Definitely made my "most practical/coolest that it actually works" list for 2024 :)
avidphantasm 2y ago
It seems like unified memory has to be the goal. This all just feels like a kludgy workaround until that happens (kind of like segmented memory in the 16-bit era).
- LorenDB 2y ago
  Is unified memory practical for a "normal" desktop/server configuration though? Apple has been doing unified memory, but they also have the GPU on the CPU die. I would be interested to know if a discrete GPU plugged into a PCIe slot would have enough latency to make unified memory impractical.
  - avidphantasm 2y ago
    It’s clearly not practical now, but that doesn’t mean it won’t be at some point.
- bsder 2y ago
  Why is RBAR insufficient? It's been pretty much supported for at least 5 years now.
  - gavinsyancey 2y ago
    GPUs benefit from extremely high memory bandwidth. RBAR helps, but it's a lot worse than a fat bus to a bunch of on-card GDDR6X. A PCIe 4.0x16 link gives 32 GB/s bandwidth; an RTX 4090 has over 1 TB/s bandwidth to its on-card memory.
    - bsder 2y ago
      Sure, but we're talking about RPC/syscall for disk and network transfers from the CPU side. Almost nothing on the CPU side can sustain 1 TB/s anyway--you can only do GPU->GPU transforms for that--and even then for very specific workloads. And the only reason you are reaching off the GPU is because you either need new data or you need the CPU to chew on something that the GPU can't manage due to branchiness.
Hydrocarb0n 2y ago
This is over my head, will it have any application or performance improvements for gpu passthrough in VMs? Or games, or make hashcat faster?
- saagarjha 2y ago
  No. This is mostly a toy rather than a production thing.
LegNeato 2y ago
If you are interested in this, you might be interested in Rust GPU: https://rust-gpu.github.io/ https://rust-gpu.github.io/

193 points by hochmartinez 2y ago

this was presented at llvm this year https://www.youtube.com/watch?v=4TxGWis1mws https://www.youtube.com/watch?v=4TxGWis1mws - it was a nice talk.

kelsey98765431 2y ago

Running normal c directly on gpu has been the dream for a long time. this looks excellent

C-programmer 2y ago
Genuinely, why? - For new code, all of the functions [here](https://libc.llvm.org/gpu/support.html#libc-gpu-support https://libc.llvm.org/gpu/support.html#libc-gpu-support) you can do without just fine. - For old code: * Your project is large enough that you are likely use using an unsupported libc function somewhere. * Your project is small enough that you would benefit from just implementing a new kernel yourself. I am biased because I avoid the C standard library even on the CPU, but this seems like a technology that raises the floor not the ceiling of what is possible.
- tredre3 2y ago
  > this seems like a technology that raises the floor not the ceiling of what is possible. In your view, how is making GPU programming easier a bad thing?
  - convolvatron 2y ago
    that's clearly not a bad thing. however encouraging people to run mutating, procedural code with explicit loops and aliasing maybe isn't the right path to get there. particularly if you just drag forward all the weird old baggage with libc and its horrible string conventions. I think any programming environment that treats a gpu as a really slow serial cpu isn't really what you want(?)
    - quotemstr 2y ago
      What if it encourages people to write parallel and functional code on CPUs? That'd be a good thing. Influence works both ways. The bigger problem is that GPUs have various platform features (shared memory, explicit cache residency and invalidation management) that CPUs sadly don't yet. Sure, you could expose these facilities via compiler intrinsics, but then you end up code that might be syntactically valid C but is alien both to CPUs and human minds
      - fc417fc802 2y ago
        > is alien both to CPUs and human minds On the contrary I would love that. The best case scenario in my mind is being able to express the native paradigms of all relevant platforms while writing a single piece of code that can then be compiled for any number of backends and dynamically retargeted between them at runtime. It would make debugging and just about everything else SO MUCH EASIER. The equivalent of being able to compile some subset of functions for both ARM and x86 and then being able to dynamically dispatch to either version at runtime, except replace ARM with a list of all the GPU ISAs that you care about.
- JonChesterfield 2y ago
  One thing this gives you is syscall on the gpu. Functions like sprintf are just blobs of userspace code, but others like fopen require support from the operating system (or whatever else the hardware needs you to do). That plumbing was decently annoying to write for the gpu. These aren't gpu kernels. They're functions to call from kernels.
  - almostgotcaught 2y ago
    > One thing this gives you is syscall on the gpu i wish people in our industry would stop (forever, completely, absolutely) using metaphors/allusions. it's a complete disservice to anyone that isn't in on the trick. it doesn't give you syscalls. that's impossible because there's no sys/os on a gpu and your actual os does not (necessarily) have any way to peer into the address space/schedular/etc of a gpu core. what it gives you is something that's working really really hard to pretend be a syscall: > Traditionally, the C library abstracts over several functions that interface with the platform’s operating system through system calls. The GPU however does not provide an operating system that can handle target dependent operations. Instead, we implemented remote procedure calls to interface with the host’s operating system while executing on a GPU. https://libc.llvm.org/gpu/rpc.html https://libc.llvm.org/gpu/rpc.html.
    - quotemstr 2y ago
      It's a matter of perspective. If you think of the GPU as a separate computer, you're right. If you think of it as a coprocessor, then the use of RPC is just an implementation detail of the system call mechanism, not a semantically different thing. When an old school 486SX delegates a floating point instruction to a physically separate 487DX coprocessor, is it executing an instruction or doing an RPC? If RPC, does the same instruction start being a real instruction when you replace your 486SX with a 486DX, with an integrated GPU? The program can't tell the difference!
      - almostgotcaught 2y ago
        > It's a matter of perspective. If you think of the GPU as a separate computer, you're right. this perspective is a function of exactly one thing: do you care about the performance of your program? if not then sure indulge in whatever abstract perspective you want ("it's magic, i just press buttons and the lights blink"). but if you don't care about perf then why are you using a GPU at all...? so for people that aren't just randomly running code on a GPU (for shits and giggles), the distinction is very significant between "syscall" and syscall. people who say these things don't program GPUs for a living. there are no abstractions unless you don't care about your program's performance (in which case why are you using a GPU at all).
- nickysielicki 2y ago
  https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/#unified_memory_after_hmm https://developer.nvidia.com/blog/simplifying-gpu-applicatio...
- deleted 2y ago
  [deleted]
- JonChesterfield 2y ago
  > Genuinely, why? > ... this seems like a technology that raises the floor not the ceiling of what is possible. The root cause reason for this project existing is to show that GPU programming is not synonymous with CUDA (or the other offloading languages). It's nominally to help people run existing code on GPUs. Disregarding that use case, it shows that GPUs can actually do things like fprintf or open sockets. This is obvious to the implementation but seems largely missed by application developers. Lots of people think GPUs can only do floating point math. Especially on an APU, where the GPU units and the CPU cores can hammer on the same memory, it is a travesty to persist with the "offloading to accelerator" model. Raw C++ isn't an especially sensible language to program GPUs in but it's workable and I think it's better than CUDA.
  - rbanffy 2y ago
    > Lots of people think GPUs can only do floating point math. IIRC, every Raspberry Pi is brought up by the GPU setting up the system before the CPU is brought out of reset and the bootloader looks for the OS. > it is a travesty to persist with the "offloading to accelerator" model. Operating systems would need to support heterogeneous processors running programs with different ISAs accessing the same pools of memory. I'd LOVE to see that. It'd be extremely convenient to have first-class processes running on the GPU MIMD cores. I'm not sure there is much research done in that space. I believe IBM mainframe OSs have something like that because programmers are exposed to the various hardware assists that run as coprocessors sharing the main memory with the OS and applications.
    - als0 2y ago
      > I'm not sure there is much research done in that space. There is. And the finest example I can think of is Barrelfish https://barrelfish.org https://barrelfish.org
      - rbanffy 2y ago
        Interesting - it resembles a network of heterogeneous systems that can share a memory space used primarily for explicit data exchange. Not quite what I was imagining, but probably much simpler to implement than a Unix where the kernel can see processes running on different ISAs on a shared memory space. I guess hardware availability is an issue, as there aren't many computers with, say, an ARM, a RISC-V, an x86, and an AMD iGPU sharing a common memory pool. OTOH, there are many where a 32-bit ARM shares the memory pool with 64-bit cores. Usually the big cores run applications while the small ARM does housekeeping or other low-latency task.
amelius 2y ago
I wonder what C would look like if CPUs would evolve into what GPUs are today. What if a CPU had assembly instructions for everything a GPU can do? Would compiler/language designers support them?
- gpderetta 2y ago
  there is no reason to wonder: https://ispc.github.io/ https://ispc.github.io/
quotemstr 2y ago
I've never understood why people say you "can't" do this or that on GPU. A GPU is made of SMs, and each SM is just a CPU with very wide SIMD pipes and very good hyperthreading. You can take one thread of a warp in a SM and do exactly the same things a CPU would do. Would you get 1/32 potential performance? Sure. But so what? Years ago, we did plenty of useful work with less than 1/32 of a modest CPU, and we can again. One of the more annoying parts of the Nvidia experience is PTX. I. I know perfectly well that your CPU/SM/whatever has a program counter. Let me manipulate it directly!
- bee_rider 2y ago
  I think people are just saying it isn’t very cost effective to use a whole GPU as 1/32 of a modest (modern?) CPU.
  - quotemstr 2y ago
    Cars are slow and inefficient in reverse gear but a car that couldn't drive in reverse would be broken.
    - bee_rider 2y ago
      You put “can’t” in quotes, so I guess you are quoting somebody, but I don’t see where the quote is from, so I’m not sure what they actually meant. But I suspect they are using “can’t” informally. Like: You can’t run a drag race in reverse. Ok, you technically could, but it would be a silly thing to do.
  - imtringued 2y ago
    Why would that matter? Adding extra CPU cores to the GPU sounds like a much dumber idea to me. You'd be wasting silicon on something that is meant to be used infrequently. Is it really that difficult to imagine 1% of your workload doing some sort of management code, except since it is running on your GPU, you can now freely intersperse those 1% every 50 microseconds without too much overhead?

Archit3ch 2y ago

No direct Metal target.

almostgotcaught 2y ago
this is an LLVM project... you want this to work on Metal, ask apple to add a Metal backend to LLVM https://github.com/llvm/llvm-project/tree/main/llvm/lib/Target https://github.com/llvm/llvm-project/tree/main/llvm/lib/Targ...
- rbanffy 2y ago
  I am surprised there isn't.
JonChesterfield 2y ago
No Intel either. The port would be easy - gpuintrin.h abstracts over the intrinsics, provide an implementation for those, write a loader in terms of opencl or whatever if you want to run the test suite. The protocol needs ordered load/store on shared memory but nothing else. I wrote a paper trying to make it clear that load/store on shmem was sufficient which doesn't seem to be considered persuasive. It's specifically designed to tolerate architectures doing slopping things with cache invalidation. It could run much faster with fetch_or / fetch_and instructions (as APUs have, but PCIe does not). It could also hang off DMA but that isn't implemented (I want to have the GPU push packets over the network without involving the x64 CPU at all).

gdiamos 2y ago

Nice to see this finally after 15 years.

amelius 2y ago

I hate libc. It's such a common cause of versioning problems on my system. Can't they just stop making new versions of it?

wbl 2y ago
Dynamic linking has some benefits, but many detractors.
- rbanffy 2y ago
  It's nice not having to recompile all your software because of a CVE impacting libc, or any other fundamental component of the system.
  - adrian_b 2y ago
    In decades of experiencing various computing environments, the amount of time wasted to fix applications broken by updates to dynamically-linked libraries has dwarfed by a few orders of magnitude the amount of time saved by not recompiling statically-linked applications because of a new library version that fixes security problems.
    - rbanffy 2y ago
      What scares me about the need to recompile statically linked binaries is that the problem is invisible until a bug hits. You don't know the statically linked library is vulnerable unless you keep track of all versions that went into that binary and almost no organization does that. DLL problems are very easy to see and very obvious when they happen. But it's been a long while since I last saw one.
- adrian_b 2y ago
  If some people like dynamically-linked applications, let them have such applications. What annoys me greatly is the existence of unavoidable libraries that have been designed intentionally to not work when linked statically, which make the life difficult for those who prefer to use only statically-linked applications. The main offending library is glibc, but there are too many other such cases.

fuhsnn 2y ago

I wonder how GPU is going to access an unknown size NULL terminated string in system RAM, the strchr() source looks like normal C++. In my minimal Vulkan GPGPU experience the data need to be bound to VkDeviceMemory to be accessible through PCI bus with compute shader, is LLVM libc runtime doing similar set-ups in the background, and if so, is it faster than glibc's hand-tuned AVX implementation?

JonChesterfield 2y ago
This is libc running on the GPU, not a libc spanning CPU and GPU. The primary obstruction to doing that is persuading people to let go of glibc. The spec of "host runs antique glibc, GPU runs some other thing that interops transparently with glibc" is a nightmare of hacks and tragedy. What would be relatively easy to put together is llvm libc running on the host x64 and also llvm libc running on the GPU. There's then the option to do things like malloc() on the GPU and free() the same pointer on the CPU. Making it genuinely seamless also involves persuading people to change what function pointers are, do some work on the ABI, and preferably move to APUs because PCIe is unhelpful. There's an uphill battle to bring people along on the journey of "program the things differently". For example, here's a thread trying to drum up enthusiasm for making function pointers into integers as that makes passing them between heterogenous architectures far easier https://discourse.llvm.org/t/rfc-function-pointers-as-integers-beyond-wasm/83208/7 https://discourse.llvm.org/t/rfc-function-pointers-as-intege....
- fuhsnn 2y ago
  >making function pointers into integers as that makes passing them between heterogenous architectures This is interesting, though function pointers are long expected to be address on binary, C-brained people like me would probably adapt to the concept of "pointer to a heterogeneous lambda object" or "shared id across heterogeneous runtimes" easier.
  - JonChesterfield 2y ago
    Yeah, I might need to take another angle at the branding / sales part, possibly with a prototype in hand. I was hopeful that wasm was sufficient prior art. Whether function pointers are absolute addresses, or relative to the start of text, or resolved through trampolines filled in by the loader, or are offsets into some table is all (nominally) invisible to the language. Integer in [0, k) has the nice feature that multiple GPUs can each index into their own lookup table containing pointers to the local code section. Or for calling into another machine - it's essentially a cheap serialisation / perfect hash done at linker time. It makes indirect calls slower, but indirect calls are slow anyway so hopefully saleable.

einpoklum 2y ago

I am pretty sure this is just a gimmick. I would not call libc code in a GPU kernel. It would mean dragging in a whole bunch of stuff I don't want, and cant control or pick-and-choose. That makes sense for regular processes on the CPU; it does _not_ make sense in code you run millions of times in GPU threads. I see people saying they've "dreamed" of this or have waited so long for this to happen... well, my friends, you should not have; and I'm afraid you're in for a disappointment.

JonChesterfield 2y ago
It uses a few mb of contiguous shared memory and periodically calling a function from a host thread. Unless you only want sprintf or similar in which case neither is needed. The unused code deadstrips pretty well. Won't help compilation time. Generally you don't want libc calls in numerical kernels doing useful stuff - the most common request was for printf as a debugging crutch, I mostly wanted mmap.
- einpoklum 2y ago
  > It uses a few mb of contiguous shared memory and periodically calling a function from a host thread. Well, if someone wants to shoot themselves in the head, then by all means... > Unless you only want sprintf or similar in which case neither is needed. > ... Generally you don't want libc calls in numerical kernels doing useful > stuff - the most common request was for printf as a debugging crutch I have actually adapted a library for that particular case: https://github.com/eyalroz/printf/ https://github.com/eyalroz/printf/ I started with a standalone printf-family implementation targetting embedded devices, and (among other things) adapted it for use also with CUDA. > I mostly wanted mmap. Does it really make sense to make a gazillion mmap calls from the threads of your GPU kernel? I mean, is it really not always better to mmap on the CPU side? At most, I might do it asynchronously using a CUDA callback or some other mechanism. But I will admit I've not had that use-case.
  - rowanG077 2y ago
    There is a ton of code that is sections of massive parallelism interspersed with sections of serial non-hot code. It makes total sense to do some syscall then and continue to the next section.

jhuber6 2y ago

Very cool to see my project posted here! The motivation behind a lot of this was to have community LLVM implementations of runtime functions normally provided by the vendor libraries (C math, printf, malloc), but if you implement one function you may as well implement them all. Realistically, the infrastructure behind this project is more relevant than the C library calls themselves. The examples in the linked documentation can be used for any arbitrary C/C++ just as well as the LLVM C library, it's simply statically linking. This is what allowed me to compile and run more complicated things like libc++ and DOOM on the GPU as well. The RPC interface can also be used to implement custom host services from the GPU, or used to communicate between any two shared memory processes.

noxa 2y ago
Just wanted to say thanks for pushing on this front! I'm not using the libc portion but the improvements to clang/llvm that allow this to work have been incredible. When I was looking a few months back the only options that felt practical for writing large amounts of device code were cuda/hip or opencl and a friend suggested I just try C _and it worked_. Definitely made my "most practical/coolest that it actually works" list for 2024 :)

avidphantasm 2y ago

It seems like unified memory has to be the goal. This all just feels like a kludgy workaround until that happens (kind of like segmented memory in the 16-bit era).

LorenDB 2y ago
Is unified memory practical for a "normal" desktop/server configuration though? Apple has been doing unified memory, but they also have the GPU on the CPU die. I would be interested to know if a discrete GPU plugged into a PCIe slot would have enough latency to make unified memory impractical.
- avidphantasm 2y ago
  It’s clearly not practical now, but that doesn’t mean it won’t be at some point.
bsder 2y ago
Why is RBAR insufficient? It's been pretty much supported for at least 5 years now.
- gavinsyancey 2y ago
  GPUs benefit from extremely high memory bandwidth. RBAR helps, but it's a lot worse than a fat bus to a bunch of on-card GDDR6X. A PCIe 4.0x16 link gives 32 GB/s bandwidth; an RTX 4090 has over 1 TB/s bandwidth to its on-card memory.
  - bsder 2y ago
    Sure, but we're talking about RPC/syscall for disk and network transfers from the CPU side. Almost nothing on the CPU side can sustain 1 TB/s anyway--you can only do GPU->GPU transforms for that--and even then for very specific workloads. And the only reason you are reaching off the GPU is because you either need new data or you need the CPU to chew on something that the GPU can't manage due to branchiness.

Hydrocarb0n 2y ago

This is over my head, will it have any application or performance improvements for gpu passthrough in VMs? Or games, or make hashcat faster?

saagarjha 2y ago
No. This is mostly a toy rather than a production thing.

LegNeato 2y ago

If you are interested in this, you might be interested in Rust GPU: https://rust-gpu.github.io/ https://rust-gpu.github.io/