4 ms·

Can anybody with a deep knowledge of the AI space, explain to me what's the real moat of CUDA ? It's clear to everybody that it's not the hardware but the soft

Can anybody with a deep knowledge of the AI space, explain to me what's the real moat of CUDA ? It's clear to everybody that it's not the hardware but the software - which is the CUDA ecosystem. I've played a bit in the past with ML, but at the level of understanding I had - training some models, tweaking things, I was using higher level libraries and as far as I know, it's pretty much an if statement in those libraries to decide which backend use. So let's suppose Intel and others does manage to implement a viable competitor - am I wrong in thinking that the transitions for many users would be seamless ? That's probably not the case for researchers and people pushing the boundaries, but for most companies, my understanding is there would be not a lot of migration costs involved ?

joe_the_user 3y ago
I see the situation as a lot like the original IBM PC wars. Originally you had the IBM PC and a bunch of "compatibles" that weren't drop-in compatible but half-assed compatible - many programs needed to be re-compiled to run on them. And other large American companies made these - they didn't expect to commodify the PC, they just wanted a small piece of a big market. The actual PC clones, pure drop-in compatibles, were made in Taiwan and they took over the market. Which is to say that large companies don't want a commodified market where prices are low and everyone competes on a level playing field - which is what "seamless transition" gets you. So that's why none of these companies are working to create that.
bdd8f1df777b 3y ago
Your understanding is correct, but the predicates are not easy at all. The amount of work going into CUDA is enormous, and NVIDIA is not standing still waiting for their competitors to catch up.
fnbr 3y ago
You need performance in the high level libraries to match, on a flops/$ basis. That’s “it”. That’s easier said than done, though. Even google’s TPUs still struggle to match H100s at flops/$, and they’re really annoying to use unless you’re using Jax.
why_only_15 3y ago
It is incorrect to say that the moat is the software. The moat is primarily the compute hardware, which is still incredibly good for the price, plus really good networking equipment. CUDA is not a significant moat for massive LLM training runs, as you can see from Anthropic moving from CUDA to Trainium (and so presumably rewriting all of their kernels to Trainium).
bartimus 3y ago
I would summarize it as follows: Nvidia has taken the bottom-up approach. From (parallel processing) hardware to their development environment developed upon it. The competition (Intel) appears to be attempting to break into the market using a top-down approach. Hoping to get some share of the inference market (using their sequential processing hardware). Basically leveraging on the innovations happening at Nvidia. CUDA will always remain one step ahead.

joe_the_user 3y ago

I see the situation as a lot like the original IBM PC wars. Originally you had the IBM PC and a bunch of "compatibles" that weren't drop-in compatible but half-assed compatible - many programs needed to be re-compiled to run on them. And other large American companies made these - they didn't expect to commodify the PC, they just wanted a small piece of a big market. The actual PC clones, pure drop-in compatibles, were made in Taiwan and they took over the market. Which is to say that large companies don't want a commodified market where prices are low and everyone competes on a level playing field - which is what "seamless transition" gets you. So that's why none of these companies are working to create that.

bdd8f1df777b 3y ago

Your understanding is correct, but the predicates are not easy at all. The amount of work going into CUDA is enormous, and NVIDIA is not standing still waiting for their competitors to catch up.

fnbr 3y ago

You need performance in the high level libraries to match, on a flops/$ basis. That’s “it”. That’s easier said than done, though. Even google’s TPUs still struggle to match H100s at flops/$, and they’re really annoying to use unless you’re using Jax.

why_only_15 3y ago

It is incorrect to say that the moat is the software. The moat is primarily the compute hardware, which is still incredibly good for the price, plus really good networking equipment. CUDA is not a significant moat for massive LLM training runs, as you can see from Anthropic moving from CUDA to Trainium (and so presumably rewriting all of their kernels to Trainium).

bartimus 3y ago

I would summarize it as follows: Nvidia has taken the bottom-up approach. From (parallel processing) hardware to their development environment developed upon it. The competition (Intel) appears to be attempting to break into the market using a top-down approach. Hoping to get some share of the inference market (using their sequential processing hardware). Basically leveraging on the innovations happening at Nvidia. CUDA will always remain one step ahead.