4 ms·
Even for large ML/LLM models, how large is the part of the code that interacts with CUDA? In other words, how much of a moat does nvidia have if a better compet
by cm2187 3y ago
Even for large ML/LLM models, how large is the part of the code that interacts with CUDA? In other words, how much of a moat does nvidia have if a better competitor comes up? My intuition is that the code itself could probably ported in a few weeks, ie if there is a moat, it doesn't consist in the CUDA implementation.
- hx8 3y agoI haven't touched CUDA or OpenCL in a decade, but here we go. These are actually pretty low level languages. The GPU has layers of memory caching. The primary challenge of SIMD programming (single instruction multiple data) is breaking up a problem (divide and conquer) in the most memory efficient way. Because "memory efficient" is highly dependent on the architecture of the card, you probably are writing code targeting a narrow band of GPUs. Sometimes you are adding code that optimizes for a specific card. How challenging is it to rewrite CUDA into an alternative? One, it requires an algorithmic understanding down to memory management. Two, it requires rewriting the entire algorithm. Three, it requires optimizing the algorithm for each card you are targeting. Finally, because Nvidia GPUs are the best, most people just code in CUDA because CUDA is better than the alternatives. It's just a good developer experience. This creates a cycle where Nvidia sells more GPUs for ML, so they can put more R&D into GPUs for ML.