4 ms·

Cool to see one of these in C, particularly if it can be binary compatible. Why not s/libreCuInit/cuInit? If you are interested in open source runtimes, tinygr

Cool to see one of these in C, particularly if it can be binary compatible. Why not s/libreCuInit/cuInit? If you are interested in open source runtimes, tinygrad has them in Python for both AMD and NVIDIA, speaking directly to the kernel through ioctls and poking the command queues. https://github.com/tinygrad/tinygrad/blob/master/tinygrad/runtime/ops_nv.py https://github.com/tinygrad/tinygrad/blob/master/tinygrad/ru... https://github.com/tinygrad/tinygrad/blob/master/tinygrad/runtime/ops_amd.py https://github.com/tinygrad/tinygrad/blob/master/tinygrad/ru...

ZoomerCretin 2y ago
Incredible! Any plans to support SASS instructions for Nvidia GPUs, or only PTX?
- georgehotz 2y ago
  We'll get there as we push deeper into assemblies. RDNA3 probably first, since it's documented and a bit simpler.
  - mike64_t 2y ago
    How do you plan on finding instruction latencies for eg. sm_89?
JonChesterfield 2y ago
That's interesting. This looks like you've bypassed the rocm userspace stack entirely. I've been looking for justification to burn libhsa.so out of the dependency graph for running llvm compiled kernels on amdgpu for ages now. I didn't expect roct to be similarly easy to drop but that's a clear sketch of how to build a statically linked freestanding x64 / gcn blob. Excellent. (I want a reference implementation of run-simple-stuff which doesn't fall over because of bugs in libhsa so that I know whatever bug I'm looking at is in my compiler / the hardware / the firmware)
- georgehotz 2y ago
  We didn't just bypass all of ROCm, we bypassed HSA! The HSA parsing MEC firmware running on the GPUs is riddled with bugs, fortunately you can bypass 90% of it using PM4, which is pretty much direct sets of the GPU registers. That's what tinygrad does. AMD's software is a really sad state. They don't have consumer GPUs in CI, they have no fuzz testing, and instead of root causing bugs they seem to just twiddle things until the application works. Between our PM4 backend and disabling CWSR, our AMD GPUs are now pretty stable.
  - JonChesterfield 2y ago
    That is a step further indeed! You're essentially running graphics shaders to do compute work. Disabling CWSR (for others following along, it's a mechanism for multiprocess time sharing) bypasses a lot of failure modes. There is a lot one cannot wisely say about the political structures which give rise to the behaviour of the rocm toolchain. Some of the implementation choices reified in the code are far divorced from what I consider reasonable, but one does not take a hatchet to other people's work. This makes fixing rocm primarily a people oriented problem to which compiler engineers are not necessarily temperamentally suited. I note that a technically inept but politically minded engineer can do extremely widespread damage in a friendly non-toxic collaborative environment. Cutting the egregiously broken parts of rocm from one's dependency graph is a solid workaround.
mike64_t 2y ago
Binary compatibility is possible, but not my main concern just yet. The CUDA api is missing length parameters left and right, often for highly problematic things such as "how long is this ELF file" and "how many parameters does this kernel need". I will definitely write wrapper headers at some point, but I don't want those hacks in the actual source code...

ZoomerCretin 2y ago

Incredible! Any plans to support SASS instructions for Nvidia GPUs, or only PTX?

georgehotz 2y ago
We'll get there as we push deeper into assemblies. RDNA3 probably first, since it's documented and a bit simpler.
- mike64_t 2y ago
  How do you plan on finding instruction latencies for eg. sm_89?

JonChesterfield 2y ago

That's interesting. This looks like you've bypassed the rocm userspace stack entirely. I've been looking for justification to burn libhsa.so out of the dependency graph for running llvm compiled kernels on amdgpu for ages now. I didn't expect roct to be similarly easy to drop but that's a clear sketch of how to build a statically linked freestanding x64 / gcn blob. Excellent. (I want a reference implementation of run-simple-stuff which doesn't fall over because of bugs in libhsa so that I know whatever bug I'm looking at is in my compiler / the hardware / the firmware)

georgehotz 2y ago
We didn't just bypass all of ROCm, we bypassed HSA! The HSA parsing MEC firmware running on the GPUs is riddled with bugs, fortunately you can bypass 90% of it using PM4, which is pretty much direct sets of the GPU registers. That's what tinygrad does. AMD's software is a really sad state. They don't have consumer GPUs in CI, they have no fuzz testing, and instead of root causing bugs they seem to just twiddle things until the application works. Between our PM4 backend and disabling CWSR, our AMD GPUs are now pretty stable.
- JonChesterfield 2y ago
  That is a step further indeed! You're essentially running graphics shaders to do compute work. Disabling CWSR (for others following along, it's a mechanism for multiprocess time sharing) bypasses a lot of failure modes. There is a lot one cannot wisely say about the political structures which give rise to the behaviour of the rocm toolchain. Some of the implementation choices reified in the code are far divorced from what I consider reasonable, but one does not take a hatchet to other people's work. This makes fixing rocm primarily a people oriented problem to which compiler engineers are not necessarily temperamentally suited. I note that a technically inept but politically minded engineer can do extremely widespread damage in a friendly non-toxic collaborative environment. Cutting the egregiously broken parts of rocm from one's dependency graph is a solid workaround.

mike64_t 2y ago

Binary compatibility is possible, but not my main concern just yet. The CUDA api is missing length parameters left and right, often for highly problematic things such as "how long is this ELF file" and "how many parameters does this kernel need". I will definitely write wrapper headers at some point, but I don't want those hacks in the actual source code...