7 ms·

Got downvoted out of view for saying this once but no less true. Absolutely a pity & a shame no one else has competed with this market dominance by Nvidia. Just

Got downvoted out of view for saying this once but no less true. Absolutely a pity & a shame no one else has competed with this market dominance by Nvidia. Just a shit world entirely that we are single vendored up. A lot of pissant shitty defenses of monopolization too. Wrong or right, this is a shit world we're in now. https://news.ycombinator.com/item?id=36304225 https://news.ycombinator.com/item?id=36304225

fragmede 3y ago
tinygrad is trying to address this problem. We'll see if it's successful.
m00x 3y ago
You can also use TPUs or other training cards. Nvidia is just the best one that's accessible. But I think you're getting downvoted since it's very off-topic.
MuffinFlavored 3y ago
it’s uh… not that serious
smoldesu 3y ago
Can hardware vendors even put their differences aside to build such a thing? We can't even build a unified open raster graphics API, and now you're asking for machine learning acceleration in that vein?
- OkayPhysicist 3y ago
  Machine learning would probably be the simpler API. If you can speak Linear Algebra, you're most of the way there.
  - smoldesu 3y ago
    You're right, and it's why projects like the ONNX runtime exist to unify vendor-specific AI accelerators. Covering the basics isn't too hard. What GP seems to be asking for is an open CUDA replacement, which is kinda like asking someone to fund a Free and Open Source cruise ship to compete with Carnival for you. You'll get somewhere with some effort, luck and good old human intuition, but Nvidia can outspend you 10:1 unless you have funding leverage from FAANG.
asynchronous 3y ago
AMD almost pulled it off but software made Nvidia take the lead again. AI being CUDA dependent really gave them an edge, in both the consumer and business market.
- deleted 3y ago
  [deleted]
- josephg 3y ago
  I have hope that AMD can close that gap over the next few years. Their hardware is already great, and the business case for investing in AI software is crazy strong. Their stock price would probably get a bump just from announcing the investment. It seems like a no brainer to hire a bunch of people to work on making PyTorch / Tensorflow on AMD become a competitive option. It’ll just take a few years.
  - raverbashing 3y ago
    I think it's been a couple of years already that we hoped better of AMD, to general disappointment
deleted 3y ago
[deleted]
reissbaker 3y ago
Apple's take on GPUs is quite different and very interesting IMO. Shared memory architecture with absolutely massive RAM (and hence VRAM) support, e.g. the new Mac Studio having 192GB RAM/VRAM which can run pretty massive models, much more so than is easily consumer-accessible even at the high end with Nvidia 4090s. It's not as fast as Nvidia, but it's not horribly far off in the latest chips. As LLM adoption grows, I wonder whether Apple's approach will start to make more sense for consumer adoption, so that you can run models on your machine without needing to pay large subscription costs for AI-powered apps (since OpenAI et al have fairly high fees). The high cost of using the APIs in my opinion is a drag on certain types of adoption in consumer apps. Llama.cpp actually first started as a way to run LLMs on Macs! At first CPU-only, but then later the first GPU driver backend added was Metal, not anything from Nvidia.
- mliker 3y ago
  In addition to the cost, OpenAI censors their models, and while that seems protective at first, if you think about it, if the model is the knowledge graph, then censor data in the graph is censoring free speech.
  - Tostino 3y ago
    I don't personally like model censorship, but you need to come up with a better argument than...whatever that is. It is entirely unconvincing, and seems like you misunderstand what free speech is.
    - mliker 3y ago
      Freedom of speech is a principle that supports the right of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction. Since the models are not suppressed to answer certain questions and indeed have been demonstrated to be biased toward one end of the political spectrum, and if you treat the model’s output as a knowledge graph, as proposed by John Schulman, one of the cofounders of OpenAI, then yes, I would say the suppression of freedom of speech is a valid argument to make. Otherwise why would there be a set of “uncensored” models that exist in the open source world? I would suggest you read about these models and think about the implications. Perhaps that’ll lead to you to reconsider your stance
deleted 3y ago
[deleted]
lachlan_gray 3y ago
With any luck projects like MLC will help close the gap https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm
AeiumNE 3y ago
I've said it before and I'll say it again. Cuda dominance is the darkest timeline. OpenCL was the utopia timeline.
- Ono-Sendai 3y ago
  OpenCL still works however :)
- smoldesu 3y ago
  Shame Apple never really invested in OpenCL. Things would look a lot different now if they hadn't abandoned it.
jiggawatts 3y ago
I wonder if it would be possible to emulate/translate CUDA to target non-NVIDIA hardware. I suspect it would be more of a legal challenge than a technical one.
- mahkeiro 3y ago
  That's what ROCm was trying to do
ygouzerh 3y ago
High technology hardware needs a lot of investment, and it's hard to gather enough resources (human, capital, market share). But AMD is coming strong, and they are trying to compete with Nvidia now. https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-saved-amd-now-she-wants-nvidias-ai-crown/?sh=598527cf1ec9 https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-s...
croes 3y ago
The others tried with OpenCL to build a more open environment, this will always lose against a single vendor tailoring it's solution for their own lineup. Think of Apples ecosystem vs Android or MS's Office-Outlook-Teams vs anything else.
ec109685 3y ago
Overview of some of the efforts to move away from cuda: https://www.semianalysis.com/p/nvidiaopenaitritonpytorch https://www.semianalysis.com/p/nvidiaopenaitritonpytorch
eur0pa 3y ago
There are far more important things to be this worried about
numlock86 3y ago
> Absolutely a pity & a shame no one else has competed with this market dominance by Nvidia. Well, at least you admit it's not Nvidia's fault. Apparently Apple, Intel and AMD don't think there's much money to grab here.
- rapsey 3y ago
  I'm sure Intel and AMD heavily regret their neglect of OpenCL since the rise of LLMs and stable diffusion.
sharikous 3y ago
WebGPU strikes me as the answer to that. Perhaps I am missing something?
- kayvr 3y ago
  It could be. But there's quite a bit of momentum behind CUDA. Plus, CUDA is just wicked fast. I wrote a WebGPU version of LLaMA inference and there's still a bit of a gap in performance between WebGPU and CUDA. Admittedly, WebGPU can't access tensor cores and I undoubtedly need to optimize further.
jeron 3y ago
looking forward to AMD MI300, hopefully will be a game changer
wyldfire 3y ago
Well, we could do a much much better job of it but in fact Qualcomm does compete with NVIDIA for use cases like this (inference). Both in mobile devices and the data center. Disclaimer: I work at Qualcomm.
- Tostino 3y ago
  What is on offer optimized for running these LLMs?
  - wyldfire 3y ago
    The hexagon NSP is reasonably well suited for running ML in general. I know it's used for some image/CV use cases and I think it will work well for language models, but maybe suboptimal for the recent large ones. This processor shows up in both snapdragon SoCs and cloud ai 100.
earhart 3y ago
IMHO, it comes down to the software. It turns out you need very different kernels for good performance on different GPUs, so OpenCL is a nice tool, but not sufficient; you need a hardware-specific kernel library. From the framework side, each integration is relatively expensive to support, so you really don’t want to invest in many of them. Without some sort of kernel API standard, you’re into a proprietary solution, and NVidia did an amazing job at investing in their software, so that’s the way things go. I think we had a pretty solid foundation for doing something smarter with PlaidML, but after we were bought by Intel, some architectural decisions and some business decisions consigned that to be a research project; I don’t know that it’s going anywhere. These days, I’d probably look into OctoML / TVM, or maybe Modular, for a better solution in this space… or just buy NVidia. (I worked a bit on Intel’s Meteor Lake VPU; it’s a lovely machine, but I’m not sure what the story will be for general framework integrations. I bet OpenVINO will run really well on it, though :-)
baq 3y ago
For me it’s doubly amazing that Intel does not exist in those discussions about alternatives that are already rare. They should write a book about how to blow up a successfully semiconductor company from the inside.
- smoldesu 3y ago
  FWIW, Intel has OpenVINO acceleration on their ARC GPU lineup. Their $300 Arc A770 outperforms the M1 Ultra by ~10% in OpenCL (which OpenVINO uses): https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks It stands to reason that Intel is making highly price-competitive hardware at the moment, but people don't talk about them as much Nvidia because they have a minuscule install base with primitive Windows drivers. I wouldn't count them out if their first showing is this impressive, though.

fragmede 3y ago

tinygrad is trying to address this problem. We'll see if it's successful.

m00x 3y ago

You can also use TPUs or other training cards. Nvidia is just the best one that's accessible. But I think you're getting downvoted since it's very off-topic.

MuffinFlavored 3y ago

it’s uh… not that serious

smoldesu 3y ago

Can hardware vendors even put their differences aside to build such a thing? We can't even build a unified open raster graphics API, and now you're asking for machine learning acceleration in that vein?

OkayPhysicist 3y ago
Machine learning would probably be the simpler API. If you can speak Linear Algebra, you're most of the way there.
- smoldesu 3y ago
  You're right, and it's why projects like the ONNX runtime exist to unify vendor-specific AI accelerators. Covering the basics isn't too hard. What GP seems to be asking for is an open CUDA replacement, which is kinda like asking someone to fund a Free and Open Source cruise ship to compete with Carnival for you. You'll get somewhere with some effort, luck and good old human intuition, but Nvidia can outspend you 10:1 unless you have funding leverage from FAANG.

asynchronous 3y ago

AMD almost pulled it off but software made Nvidia take the lead again. AI being CUDA dependent really gave them an edge, in both the consumer and business market.

deleted 3y ago
[deleted]
josephg 3y ago
I have hope that AMD can close that gap over the next few years. Their hardware is already great, and the business case for investing in AI software is crazy strong. Their stock price would probably get a bump just from announcing the investment. It seems like a no brainer to hire a bunch of people to work on making PyTorch / Tensorflow on AMD become a competitive option. It’ll just take a few years.
- raverbashing 3y ago
  I think it's been a couple of years already that we hoped better of AMD, to general disappointment

deleted 3y ago

[deleted]

reissbaker 3y ago

Apple's take on GPUs is quite different and very interesting IMO. Shared memory architecture with absolutely massive RAM (and hence VRAM) support, e.g. the new Mac Studio having 192GB RAM/VRAM which can run pretty massive models, much more so than is easily consumer-accessible even at the high end with Nvidia 4090s. It's not as fast as Nvidia, but it's not horribly far off in the latest chips. As LLM adoption grows, I wonder whether Apple's approach will start to make more sense for consumer adoption, so that you can run models on your machine without needing to pay large subscription costs for AI-powered apps (since OpenAI et al have fairly high fees). The high cost of using the APIs in my opinion is a drag on certain types of adoption in consumer apps. Llama.cpp actually first started as a way to run LLMs on Macs! At first CPU-only, but then later the first GPU driver backend added was Metal, not anything from Nvidia.

mliker 3y ago
In addition to the cost, OpenAI censors their models, and while that seems protective at first, if you think about it, if the model is the knowledge graph, then censor data in the graph is censoring free speech.
- Tostino 3y ago
  I don't personally like model censorship, but you need to come up with a better argument than...whatever that is. It is entirely unconvincing, and seems like you misunderstand what free speech is.
  - mliker 3y ago
    Freedom of speech is a principle that supports the right of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction. Since the models are not suppressed to answer certain questions and indeed have been demonstrated to be biased toward one end of the political spectrum, and if you treat the model’s output as a knowledge graph, as proposed by John Schulman, one of the cofounders of OpenAI, then yes, I would say the suppression of freedom of speech is a valid argument to make. Otherwise why would there be a set of “uncensored” models that exist in the open source world? I would suggest you read about these models and think about the implications. Perhaps that’ll lead to you to reconsider your stance

deleted 3y ago

[deleted]

lachlan_gray 3y ago

With any luck projects like MLC will help close the gap https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm

AeiumNE 3y ago

I've said it before and I'll say it again. Cuda dominance is the darkest timeline. OpenCL was the utopia timeline.

Ono-Sendai 3y ago
OpenCL still works however :)
smoldesu 3y ago
Shame Apple never really invested in OpenCL. Things would look a lot different now if they hadn't abandoned it.

jiggawatts 3y ago

I wonder if it would be possible to emulate/translate CUDA to target non-NVIDIA hardware. I suspect it would be more of a legal challenge than a technical one.

mahkeiro 3y ago
That's what ROCm was trying to do

ygouzerh 3y ago

High technology hardware needs a lot of investment, and it's hard to gather enough resources (human, capital, market share). But AMD is coming strong, and they are trying to compete with Nvidia now. https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-saved-amd-now-she-wants-nvidias-ai-crown/?sh=598527cf1ec9 https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-s...

croes 3y ago

The others tried with OpenCL to build a more open environment, this will always lose against a single vendor tailoring it's solution for their own lineup. Think of Apples ecosystem vs Android or MS's Office-Outlook-Teams vs anything else.

ec109685 3y ago

Overview of some of the efforts to move away from cuda: https://www.semianalysis.com/p/nvidiaopenaitritonpytorch https://www.semianalysis.com/p/nvidiaopenaitritonpytorch

eur0pa 3y ago

There are far more important things to be this worried about

numlock86 3y ago

> Absolutely a pity & a shame no one else has competed with this market dominance by Nvidia. Well, at least you admit it's not Nvidia's fault. Apparently Apple, Intel and AMD don't think there's much money to grab here.

rapsey 3y ago
I'm sure Intel and AMD heavily regret their neglect of OpenCL since the rise of LLMs and stable diffusion.

sharikous 3y ago

WebGPU strikes me as the answer to that. Perhaps I am missing something?

kayvr 3y ago
It could be. But there's quite a bit of momentum behind CUDA. Plus, CUDA is just wicked fast. I wrote a WebGPU version of LLaMA inference and there's still a bit of a gap in performance between WebGPU and CUDA. Admittedly, WebGPU can't access tensor cores and I undoubtedly need to optimize further.

jeron 3y ago

looking forward to AMD MI300, hopefully will be a game changer

wyldfire 3y ago

Well, we could do a much much better job of it but in fact Qualcomm does compete with NVIDIA for use cases like this (inference). Both in mobile devices and the data center. Disclaimer: I work at Qualcomm.

Tostino 3y ago
What is on offer optimized for running these LLMs?
- wyldfire 3y ago
  The hexagon NSP is reasonably well suited for running ML in general. I know it's used for some image/CV use cases and I think it will work well for language models, but maybe suboptimal for the recent large ones. This processor shows up in both snapdragon SoCs and cloud ai 100.

earhart 3y ago

IMHO, it comes down to the software. It turns out you need very different kernels for good performance on different GPUs, so OpenCL is a nice tool, but not sufficient; you need a hardware-specific kernel library. From the framework side, each integration is relatively expensive to support, so you really don’t want to invest in many of them. Without some sort of kernel API standard, you’re into a proprietary solution, and NVidia did an amazing job at investing in their software, so that’s the way things go. I think we had a pretty solid foundation for doing something smarter with PlaidML, but after we were bought by Intel, some architectural decisions and some business decisions consigned that to be a research project; I don’t know that it’s going anywhere. These days, I’d probably look into OctoML / TVM, or maybe Modular, for a better solution in this space… or just buy NVidia. (I worked a bit on Intel’s Meteor Lake VPU; it’s a lovely machine, but I’m not sure what the story will be for general framework integrations. I bet OpenVINO will run really well on it, though :-)

baq 3y ago

For me it’s doubly amazing that Intel does not exist in those discussions about alternatives that are already rare. They should write a book about how to blow up a successfully semiconductor company from the inside.

smoldesu 3y ago
FWIW, Intel has OpenVINO acceleration on their ARC GPU lineup. Their $300 Arc A770 outperforms the M1 Ultra by ~10% in OpenCL (which OpenVINO uses): https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks It stands to reason that Intel is making highly price-competitive hardware at the moment, but people don't talk about them as much Nvidia because they have a minuscule install base with primitive Windows drivers. I wouldn't count them out if their first showing is this impressive, though.