I haven't touched CUDA or OpenCL in a decade, but here we go. These are actually pretty low level languages. The GPU has layers of memory caching. The primar

I haven't touched CUDA or OpenCL in a decade, but here we go. These are actually pretty low level languages. The GPU has layers of memory caching. The primary challenge of SIMD programming (single instruction multiple data) is breaking up a problem (divide and conquer) in the most memory efficient way. Because "memory efficient" is highly dependent on the architecture of the card, you probably are writing code targeting a narrow band of GPUs. Sometimes you are adding code that optimizes for a specific card. How challenging is it to rewrite CUDA into an alternative? One, it requires an algorithmic understanding down to memory management. Two, it requires rewriting the entire algorithm. Three, it requires optimizing the algorithm for each card you are targeting. Finally, because Nvidia GPUs are the best, most people just code in CUDA because CUDA is better than the alternatives. It's just a good developer experience. This creates a cycle where Nvidia sells more GPUs for ML, so they can put more R&D into GPUs for ML.