3 ms·
We didn't just bypass all of ROCm, we bypassed HSA! The HSA parsing MEC firmware running on the GPUs is riddled with bugs, fortunately you can bypass 90% of it
by georgehotz 2y ago
We didn't just bypass all of ROCm, we bypassed HSA!
The HSA parsing MEC firmware running on the GPUs is riddled with bugs, fortunately you can bypass 90% of it using PM4, which is pretty much direct sets of the GPU registers. That's what tinygrad does.
AMD's software is a really sad state. They don't have consumer GPUs in CI, they have no fuzz testing, and instead of root causing bugs they seem to just twiddle things until the application works.
Between our PM4 backend and disabling CWSR, our AMD GPUs are now pretty stable.
- JonChesterfield 2y agoThat is a step further indeed! You're essentially running graphics shaders to do compute work. Disabling CWSR (for others following along, it's a mechanism for multiprocess time sharing) bypasses a lot of failure modes. There is a lot one cannot wisely say about the political structures which give rise to the behaviour of the rocm toolchain. Some of the implementation choices reified in the code are far divorced from what I consider reasonable, but one does not take a hatchet to other people's work. This makes fixing rocm primarily a people oriented problem to which compiler engineers are not necessarily temperamentally suited. I note that a technically inept but politically minded engineer can do extremely widespread damage in a friendly non-toxic collaborative environment. Cutting the egregiously broken parts of rocm from one's dependency graph is a solid workaround.