3 ms·

Could somebody explain to me what is the moat that prevents somebody else from taking on Nvidia? OK, they make GPUs. Why somebody else can't make GPUs? AMD took

by smsm42 3y ago

Could somebody explain to me what is the moat that prevents somebody else from taking on Nvidia? OK, they make GPUs. Why somebody else can't make GPUs? AMD took on Intel, Google took on Microsoft and Apple - is GPU market somehow harder to enter? I mean, we're talking piles of money from the looks of it - Amazon and Google do a ton of stuff, and NVidia only does (roughly speaking) one thing - and that thing is not even directly consumed by the customers, so "it's hard to migrate" shouldn't be a huge deal since it's not the customers that would be doing the migration. So what are the barriers that make Nvidia unique?

alecco 3y ago
GPUs are very complex. Nvidia pioneers in everything related to GPUs. A rich software stack, the most sophisticated Tensor Cores, and bleeding edge features like 8 bit floating point (FP8) support. And they are working with FP4 next. This matters because by halving the data size it almost doubles the flops (see Hopper specs [1]). The compute is so powerful it creates bottlenecks in data loading. So they have SXM, Nvlink, and since Hopper smart data async load to Tensor Cores (TMA). It's so advanced the ML software hasn't caught up yet. And it's not trivial to tile and schedule properly at these levels. (See FlashAttention [2]) I wouldn't be surprised if they stay on Hopper for a while and just crank up the bandwidth and bundle more GPUs together. They already released H100 NVL which is basically 2 H100s. And the H200 with faster High Bandwidth Memory (v3). AMD and Intel are way behind and have nothing even remotely close in planning. [1] https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor... [2] https://crfm.stanford.edu/2023/07/17/flash2.html https://crfm.stanford.edu/2023/07/17/flash2.html
Moldoteck 3y ago
nvidia is unique because of cuda and all ml software written/optimized for it. AMD can add cuda support too (and they started working in this direction) but they'll not be able to out-compete nvidia, with this strategy the'll stay at 2'nd place (by buying amd cards for inference you are by default less competitive as business doing ML compared to those that can buy nvidia) and will just strengthen nvidia's cuda moat. Imo it's less about money and more about a ton software that was optimized for nvidia cards and software that was built on top of that optimized software, it can't be easily ported to run as efficient on AMD, it's tons of code and it's unlikely devs would just port it + port all the new stuff that is researched at exponential scale.

by smsm42 3y ago

alecco 3y ago

GPUs are very complex. Nvidia pioneers in everything related to GPUs. A rich software stack, the most sophisticated Tensor Cores, and bleeding edge features like 8 bit floating point (FP8) support. And they are working with FP4 next. This matters because by halving the data size it almost doubles the flops (see Hopper specs [1]). The compute is so powerful it creates bottlenecks in data loading. So they have SXM, Nvlink, and since Hopper smart data async load to Tensor Cores (TMA). It's so advanced the ML software hasn't caught up yet. And it's not trivial to tile and schedule properly at these levels. (See FlashAttention [2]) I wouldn't be surprised if they stay on Hopper for a while and just crank up the bandwidth and bundle more GPUs together. They already released H100 NVL which is basically 2 H100s. And the H200 with faster High Bandwidth Memory (v3). AMD and Intel are way behind and have nothing even remotely close in planning. [1] https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor... [2] https://crfm.stanford.edu/2023/07/17/flash2.html https://crfm.stanford.edu/2023/07/17/flash2.html

Moldoteck 3y ago

nvidia is unique because of cuda and all ml software written/optimized for it. AMD can add cuda support too (and they started working in this direction) but they'll not be able to out-compete nvidia, with this strategy the'll stay at 2'nd place (by buying amd cards for inference you are by default less competitive as business doing ML compared to those that can buy nvidia) and will just strengthen nvidia's cuda moat. Imo it's less about money and more about a ton software that was optimized for nvidia cards and software that was built on top of that optimized software, it can't be easily ported to run as efficient on AMD, it's tons of code and it's unlikely devs would just port it + port all the new stuff that is researched at exponential scale.