9 ms·

Nvidia's CUDA Monopoly

105 points by ingve 3y ago

DeathArrow 3y ago
>software companies don’t have the hardware capabilities, and vice versa Nobody stops a hardware company to hire software developers. The other way around would be harder.
- zepmck 3y ago
  Software developers, especially for certain type of components, e.g. compilers, are NOT easy to find out.
  - dist-epoch 3y ago
    There's no such thing as hard to find. Only "I don't want to pay what they ask". Which is weird for a hardware company since you don't need that many and you only write the driver once, but you sell it with millions of devices.
  - pjmlp 3y ago
    They can start on GCC and LLVM mailing lists.
- amelius 3y ago
  Writing VHDL/Verilog is much like writing software, however.
  - deleted 3y ago
    [deleted]
- zgluck 3y ago
  Hardware companies often find it very hard to competently build software. They tend to lack appreciation/understanding of software in the first place (hardware-first thinking) leading to underinvestments. It's also hard for them to identify great software people - at all levels, starting from the CEO/board.
vardump 3y ago
I can't understand why AMD leaves so much on the table. Of course their ROCm efforts have intensified, but there's still a long way to go.
- DeathArrow 3y ago
  I wonder if ROCm is 100% CUDA compatible.
  - MayeulC 3y ago
    No. They want you to target HIP, which is compatible with both CUDA and ROCm. https://rocm.docs.amd.com/projects/HIP/en/latest/index.html https://rocm.docs.amd.com/projects/HIP/en/latest/index.html
    - Kelteseth 3y ago
      And with HIP now supporting Windows[0] it could be a game changer. I would love to see it working with llama.cpp[1]. I tried HIP via Blender, but it did not work for my 6600XT, it still used my CPU... [0] https://github.com/RadeonOpenCompute/ROCm/issues/666#issuecomment-1655470437 https://github.com/RadeonOpenCompute/ROCm/issues/666#issueco... [1] https://github.com/ggerganov/llama.cpp/discussions/915#discussioncomment-6626936 https://github.com/ggerganov/llama.cpp/discussions/915#discu...
  - Ampersander 3y ago
    It's not 100%. CUDA for example has math functions with special rounding modes, but HIP does not.
  - pjmlp 3y ago
    Not at all, as it doesn't cover all CUDA use cases.
- nicolaslem 3y ago
  Because AMD is tiny in comparison and they are fighting titans on the CPU and GPU fronts, meaning they have to pick their battles. They decided to fight Nvidia on the gaming segment and let them win productivity. It seemed like a good strategy a few years ago when GPU were unobtainium, not so much today.
eddiewithzato 3y ago
it’s all about investment, nvidia invested hard and got rewarded. Similar to how the iPhone disrupted the marketed. I don’t see an open source solution winning unless a company can invest in the development
DoingIsLearning 3y ago
Does anyone have more inside knowledge from OpenAI or AMD on AMDGPU support for Triton? I see this: https://github.com/openai/triton/issues/1073 https://github.com/openai/triton/issues/1073 But it's not clear to me if we will see AMD GPUs as first class citizens for pytorch in the future?
zepmck 3y ago
The main reasons of "monopoly" are long-term vision and enormous effort spent in software development. AMD is lacking behind because they have not yet seriously invested on their own software stack.
- yieldcrv 3y ago
  monopoly isn't a bad word here and doesn't need quotes. In America and major markets, a monopoly is fine if that's what the market chooses, anticompetitve practices and abusing market position is what gets at least put under scrutiny. They may be limited in purchasing additional companies to expand or maintain the market position, but there is no need to shift perception about what is it because that's not regulated or under threat of regulatory scrutiny simply from providing a better service.
  - DeathArrow 3y ago
    So you are fine paying 10x what's worth because the monopoly?
    - sokoloff 3y ago
      Other than government mandates, I don’t pay for things that cost more than they’re worth to me. I just don’t buy them and I think most people have a similar approach.
    - butlerm 3y ago
      No one buys any product that costs more than it is worth to them. If value is lower than cost sales tend to go to zero. The continued market for a product even in the midst of shortage indicates that someone somewhere values the product more than its market price. Otherwise the price would drop until demand converged with supply.
      - Affric 3y ago
        Sure, but often the costs of high prices of inelastic goods are externalised. People excluded from housing, banking, energy, education, and health markets necessarily turn to various anti social activities which create costs for society… and those who are making real profits rarely end up picking up the bill for these. Though I don’t believe this line of reasoning applies to CUDA/GPUs (yet).
kuschkufan 3y ago
Rather poor piece. Not even mentioning the competition (that's trying to catch up, yes). Sometimes it sounds more like gushing than anything else, always the best, consistently sold out, etc etc.
andrewstuart 3y ago
Nvidias CUDA monopoly will end itself. Nvidia is pushing upmarket, focusing on data center products it can charge huge amounts for. As Nvidia pushes upmarket, the traditional computing market forces will come into play. These forces have played out for 50 years. Users will seek cheap and available GPUs and they’ll find a way to get the job done with them. At the moment, people say they can’t use retail GPUs because of RAM constraints. This will change however. Through necessity, software will be developed that gets the job done on consumer grade GPUs. It might be open source. It might be AMD or Intel software. This has always been the way with computers…. innovation at the low end will foil the plans of IBM to control the entire market. Oops did I say IBM? I meant Nvidia. Nvidia has the chance to own everything long term, but monopolists can’t help but become greedy and become their own worst enemy. Nvidia is milking its customers for huge, huge profits. The customers will find another way, this Nvidia will have indirectly created its true competition. If Intel and AMD want to defeat Nvidia they need to not play Nvidias game of going for the high end and turning its nose up at the low end. AMD and Intel need to produce the lowest cost, most powerful GPUs they can. They also need to attack Nvidia where it hurts. Nvidia has artificial constraints on its GPUs. Through drivers it prevents certain uses so that customers are forced to use high end data center GPUs. Where ever Nvidia has artificial constraints, AMD and Intel need to NOT have those constraints. Where Nvidia is closed source, Intel and AMD need to be open source. Nvidias dominance won’t end, but it’s monopoly will, and viable competition will form simply because Nvidia is so anti consumer, and in the computing game consumers are very resourceful. What’s needed is a bunch of smart programmers who know how to squeeze usable AI out of a 24 gigabyte AMD 7900 XTX.
- mschuster91 3y ago
  > What’s needed is a bunch of smart programmers who know how to squeeze usable AI out of a 24 gigabyte AMD 7900 XTX. The first thing that's needed is for AMD to wake up and invest into its tooling stack.
  - PaywallBuster 3y ago
    That's the article we need What's AMD tools look like and what would it take to catch up with CUDA and I guess it does not depend on AMD alone, the cloud providers trying to get alternatives to Nvidia could be investing into this too
physicsguy 3y ago
Until there are core libraries that support AMD and other GPU platforms, NVidia will continue to dominate. I was really encouraged to see this the other day: https://news.ycombinator.com/item?id=36968273 https://news.ycombinator.com/item?id=36968273 Because the AMD GPU FFT library isn't feature complete compared to cuFFT or even FFTW.
andrewstuart 3y ago
I have a copy of Byte magazine with the front cover that says “Is Unix dead?”. Back then, everyone thought Windows NT had “won”, and it was game over for every other operating system. Same thing here. Nvidia might look strong now but all it takes is an open source project that competes effectively to change everything.
- DeathArrow 3y ago
  >I have a copy of Byte magazine with the front cover that says “Is Unix dead?”. For practical purposes it is. Bur it's not Windows NT that killed it, it's Linux.
  - deleted 3y ago
    [deleted]
  - worrycue 3y ago
    You miss his point. NT was thought to have “won” and their position seem unassailable but the unexpected happened and their reign in the server world was short lived.
    - pjmlp 3y ago
      It certainly won the workstation market, moreso with the ongoing upgradability issues in the Mac Pro.
  - rcme 3y ago
    Aren’t macOS and iOS Unix-based? Unix is one of the most widely used operating systems in the world right now.
    - pjmlp 3y ago
      macOS yes, iOS not really, you won't go far using only UNIX for iOS apps. The actually reality is POSIX flavoured OSes are most widely used operating systems in the world, for various levels of what means to be POSIX, and to what level they expose it. "Transcending POSIX: The End of an Era?" https://www.usenix.org/publications/loginonline/transcending-posix-end-era https://www.usenix.org/publications/loginonline/transcending...
fulafel 3y ago
GPUs are reliving the bad old days style proprietary tools & ISAs world. Which the GP computing world broke out of with the advent of multi vendor ISA compatibility in commodity CPUs (x86, later others) and open source compilers (GCC, later others). Before that the sw stacks were buggy, vendor locked, expensive, were gatekeeping programming langugaes development, very similarly as now in GPUs.
- pjmlp 3y ago
  That never changed in the GPU world, because contrary to urban myths, game consoles never really supported GL, only inspired variants of it. Not to mention extension spaghetti with proprietary features anyway.
ai_ja_nai 3y ago
I remember how it was in 2008 on Linux desktops/laptops: AMD laughed straight in the face of their Linux customers. NVidia did support them, although at its own rules (closed source). When the need to run scientific computation over GPUs arouse, NVidia was there, ready. AMD got what it deserved for ignoring a market of scientists and researchers
DrNosferatu 3y ago
No mention of OpenCL / VK compute?
- pjmlp 3y ago
  When they are able to handle C++, Fortran, Julia, Python JIT, ...., provide graphical debugging experience for GPGPU code, then maybe it will matter.
Havoc 3y ago
I’m just amazed amd has produced so sort of working shim for their stuff that enables PyTorch use. Even if it’s slow and rough. Rocm seems perpetually imminent…
PAPPPmAc 3y ago
I don't see in the article or thread, anyone else remember when in ~2012 PGI (Portland Group, who at the time were a subsidiary of STMicroelectronics), a long time builder of high-performance compiler tooling was showing a new round of platform-independent parallel accelerator tooling including teasing a credible independent CUDA implementation that could target non-Nvidia platforms, then Nvidia bought them in 2013 and it disappeared from the face of the earth in favor of "we're excited to be working on CUDA Support for FORTRAN"? ...Yeah. Nvidia seems well aware that the "There is existing code many users want to run written in CUDA, and you can only run CUDA code on an Nvidia part" situation is their competitive advantage. ATI/AMD's failure to settle on a stable GPGPU toolchain (CTM/THIN/Brook+, Stream, ROCm with OpenCL, HIP, and perpetually broken CUDA compat...) and OpenCL's ugly boilerplate gave them an opportunity to get that core set of lock-in software, and they're not giving it up without a fight.
segmondy 3y ago
the software is important, but the hardware is much more so. if AMD could make performant GPU that had solid drivers, the world would rally and port all the major ML libraries to it over night. CUDA is not the moat you think it is. Nvidia is going to shoot itself on the feet with how they have been handicapping their latest GPUs. Unfortunately, it doesn't look like AMD is prime to compete.

105 points by ingve 3y ago

>software companies don’t have the hardware capabilities, and vice versa Nobody stops a hardware company to hire software developers. The other way around would be harder.

zepmck 3y ago
Software developers, especially for certain type of components, e.g. compilers, are NOT easy to find out.
- dist-epoch 3y ago
  There's no such thing as hard to find. Only "I don't want to pay what they ask". Which is weird for a hardware company since you don't need that many and you only write the driver once, but you sell it with millions of devices.
- pjmlp 3y ago
  They can start on GCC and LLVM mailing lists.
amelius 3y ago
Writing VHDL/Verilog is much like writing software, however.
- deleted 3y ago
  [deleted]
zgluck 3y ago
Hardware companies often find it very hard to competently build software. They tend to lack appreciation/understanding of software in the first place (hardware-first thinking) leading to underinvestments. It's also hard for them to identify great software people - at all levels, starting from the CEO/board.

vardump 3y ago

I can't understand why AMD leaves so much on the table. Of course their ROCm efforts have intensified, but there's still a long way to go.

DeathArrow 3y ago
I wonder if ROCm is 100% CUDA compatible.
- MayeulC 3y ago
  No. They want you to target HIP, which is compatible with both CUDA and ROCm. https://rocm.docs.amd.com/projects/HIP/en/latest/index.html https://rocm.docs.amd.com/projects/HIP/en/latest/index.html
  - Kelteseth 3y ago
    And with HIP now supporting Windows[0] it could be a game changer. I would love to see it working with llama.cpp[1]. I tried HIP via Blender, but it did not work for my 6600XT, it still used my CPU... [0] https://github.com/RadeonOpenCompute/ROCm/issues/666#issuecomment-1655470437 https://github.com/RadeonOpenCompute/ROCm/issues/666#issueco... [1] https://github.com/ggerganov/llama.cpp/discussions/915#discussioncomment-6626936 https://github.com/ggerganov/llama.cpp/discussions/915#discu...
- Ampersander 3y ago
  It's not 100%. CUDA for example has math functions with special rounding modes, but HIP does not.
- pjmlp 3y ago
  Not at all, as it doesn't cover all CUDA use cases.
nicolaslem 3y ago
Because AMD is tiny in comparison and they are fighting titans on the CPU and GPU fronts, meaning they have to pick their battles. They decided to fight Nvidia on the gaming segment and let them win productivity. It seemed like a good strategy a few years ago when GPU were unobtainium, not so much today.

eddiewithzato 3y ago

it’s all about investment, nvidia invested hard and got rewarded. Similar to how the iPhone disrupted the marketed. I don’t see an open source solution winning unless a company can invest in the development

DoingIsLearning 3y ago

Does anyone have more inside knowledge from OpenAI or AMD on AMDGPU support for Triton? I see this: https://github.com/openai/triton/issues/1073 https://github.com/openai/triton/issues/1073 But it's not clear to me if we will see AMD GPUs as first class citizens for pytorch in the future?

zepmck 3y ago

The main reasons of "monopoly" are long-term vision and enormous effort spent in software development. AMD is lacking behind because they have not yet seriously invested on their own software stack.

yieldcrv 3y ago
monopoly isn't a bad word here and doesn't need quotes. In America and major markets, a monopoly is fine if that's what the market chooses, anticompetitve practices and abusing market position is what gets at least put under scrutiny. They may be limited in purchasing additional companies to expand or maintain the market position, but there is no need to shift perception about what is it because that's not regulated or under threat of regulatory scrutiny simply from providing a better service.
- DeathArrow 3y ago
  So you are fine paying 10x what's worth because the monopoly?
  - sokoloff 3y ago
    Other than government mandates, I don’t pay for things that cost more than they’re worth to me. I just don’t buy them and I think most people have a similar approach.
  - butlerm 3y ago
    No one buys any product that costs more than it is worth to them. If value is lower than cost sales tend to go to zero. The continued market for a product even in the midst of shortage indicates that someone somewhere values the product more than its market price. Otherwise the price would drop until demand converged with supply.
    - Affric 3y ago
      Sure, but often the costs of high prices of inelastic goods are externalised. People excluded from housing, banking, energy, education, and health markets necessarily turn to various anti social activities which create costs for society… and those who are making real profits rarely end up picking up the bill for these. Though I don’t believe this line of reasoning applies to CUDA/GPUs (yet).

kuschkufan 3y ago

Rather poor piece. Not even mentioning the competition (that's trying to catch up, yes). Sometimes it sounds more like gushing than anything else, always the best, consistently sold out, etc etc.

andrewstuart 3y ago

Nvidias CUDA monopoly will end itself. Nvidia is pushing upmarket, focusing on data center products it can charge huge amounts for. As Nvidia pushes upmarket, the traditional computing market forces will come into play. These forces have played out for 50 years. Users will seek cheap and available GPUs and they’ll find a way to get the job done with them. At the moment, people say they can’t use retail GPUs because of RAM constraints. This will change however. Through necessity, software will be developed that gets the job done on consumer grade GPUs. It might be open source. It might be AMD or Intel software. This has always been the way with computers…. innovation at the low end will foil the plans of IBM to control the entire market. Oops did I say IBM? I meant Nvidia. Nvidia has the chance to own everything long term, but monopolists can’t help but become greedy and become their own worst enemy. Nvidia is milking its customers for huge, huge profits. The customers will find another way, this Nvidia will have indirectly created its true competition. If Intel and AMD want to defeat Nvidia they need to not play Nvidias game of going for the high end and turning its nose up at the low end. AMD and Intel need to produce the lowest cost, most powerful GPUs they can. They also need to attack Nvidia where it hurts. Nvidia has artificial constraints on its GPUs. Through drivers it prevents certain uses so that customers are forced to use high end data center GPUs. Where ever Nvidia has artificial constraints, AMD and Intel need to NOT have those constraints. Where Nvidia is closed source, Intel and AMD need to be open source. Nvidias dominance won’t end, but it’s monopoly will, and viable competition will form simply because Nvidia is so anti consumer, and in the computing game consumers are very resourceful. What’s needed is a bunch of smart programmers who know how to squeeze usable AI out of a 24 gigabyte AMD 7900 XTX.

mschuster91 3y ago
> What’s needed is a bunch of smart programmers who know how to squeeze usable AI out of a 24 gigabyte AMD 7900 XTX. The first thing that's needed is for AMD to wake up and invest into its tooling stack.
- PaywallBuster 3y ago
  That's the article we need What's AMD tools look like and what would it take to catch up with CUDA and I guess it does not depend on AMD alone, the cloud providers trying to get alternatives to Nvidia could be investing into this too

physicsguy 3y ago

Until there are core libraries that support AMD and other GPU platforms, NVidia will continue to dominate. I was really encouraged to see this the other day: https://news.ycombinator.com/item?id=36968273 https://news.ycombinator.com/item?id=36968273 Because the AMD GPU FFT library isn't feature complete compared to cuFFT or even FFTW.

andrewstuart 3y ago

I have a copy of Byte magazine with the front cover that says “Is Unix dead?”. Back then, everyone thought Windows NT had “won”, and it was game over for every other operating system. Same thing here. Nvidia might look strong now but all it takes is an open source project that competes effectively to change everything.

DeathArrow 3y ago
>I have a copy of Byte magazine with the front cover that says “Is Unix dead?”. For practical purposes it is. Bur it's not Windows NT that killed it, it's Linux.
- deleted 3y ago
  [deleted]
- worrycue 3y ago
  You miss his point. NT was thought to have “won” and their position seem unassailable but the unexpected happened and their reign in the server world was short lived.
  - pjmlp 3y ago
    It certainly won the workstation market, moreso with the ongoing upgradability issues in the Mac Pro.
- rcme 3y ago
  Aren’t macOS and iOS Unix-based? Unix is one of the most widely used operating systems in the world right now.
  - pjmlp 3y ago
    macOS yes, iOS not really, you won't go far using only UNIX for iOS apps. The actually reality is POSIX flavoured OSes are most widely used operating systems in the world, for various levels of what means to be POSIX, and to what level they expose it. "Transcending POSIX: The End of an Era?" https://www.usenix.org/publications/loginonline/transcending-posix-end-era https://www.usenix.org/publications/loginonline/transcending...

fulafel 3y ago

GPUs are reliving the bad old days style proprietary tools & ISAs world. Which the GP computing world broke out of with the advent of multi vendor ISA compatibility in commodity CPUs (x86, later others) and open source compilers (GCC, later others). Before that the sw stacks were buggy, vendor locked, expensive, were gatekeeping programming langugaes development, very similarly as now in GPUs.

pjmlp 3y ago
That never changed in the GPU world, because contrary to urban myths, game consoles never really supported GL, only inspired variants of it. Not to mention extension spaghetti with proprietary features anyway.

ai_ja_nai 3y ago

I remember how it was in 2008 on Linux desktops/laptops: AMD laughed straight in the face of their Linux customers. NVidia did support them, although at its own rules (closed source). When the need to run scientific computation over GPUs arouse, NVidia was there, ready. AMD got what it deserved for ignoring a market of scientists and researchers

DrNosferatu 3y ago

No mention of OpenCL / VK compute?

pjmlp 3y ago
When they are able to handle C++, Fortran, Julia, Python JIT, ...., provide graphical debugging experience for GPGPU code, then maybe it will matter.

Havoc 3y ago

I’m just amazed amd has produced so sort of working shim for their stuff that enables PyTorch use. Even if it’s slow and rough. Rocm seems perpetually imminent…

PAPPPmAc 3y ago

I don't see in the article or thread, anyone else remember when in ~2012 PGI (Portland Group, who at the time were a subsidiary of STMicroelectronics), a long time builder of high-performance compiler tooling was showing a new round of platform-independent parallel accelerator tooling including teasing a credible independent CUDA implementation that could target non-Nvidia platforms, then Nvidia bought them in 2013 and it disappeared from the face of the earth in favor of "we're excited to be working on CUDA Support for FORTRAN"? ...Yeah. Nvidia seems well aware that the "There is existing code many users want to run written in CUDA, and you can only run CUDA code on an Nvidia part" situation is their competitive advantage. ATI/AMD's failure to settle on a stable GPGPU toolchain (CTM/THIN/Brook+, Stream, ROCm with OpenCL, HIP, and perpetually broken CUDA compat...) and OpenCL's ugly boilerplate gave them an opportunity to get that core set of lock-in software, and they're not giving it up without a fight.

segmondy 3y ago

the software is important, but the hardware is much more so. if AMD could make performant GPU that had solid drivers, the world would rally and port all the major ML libraries to it over night. CUDA is not the moat you think it is. Nvidia is going to shoot itself on the feet with how they have been handicapping their latest GPUs. Unfortunately, it doesn't look like AMD is prime to compete.