6 ms·

I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance

I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing, LLM's as a whole are becoming commoditized, and compute demand for training will drop off a cliff sooner than people expect.

epolanski 2y ago
I am a partial believer that the real race for many tech players is actually AGI and ASI later and till the problem is solved the hardware arm race will keep being part of it. Not only big tech is part of it but billion dollars startups are popping everywhere from China to US and Middle East.
- chrisdbanks 2y ago
  Unless we hit another AI winter. We might get to the point where the hardware just can't give better returns and have to wait another 20 years for the next leap forward. We're still orders of magnitude away from the human brain.
deleted 2y ago
[deleted]
nextworddev 2y ago
It’s actually about training, not inference. You can’t do training on commodity gpus but yeah once someone figures that out, nvdia could crash
- gradus_ad 2y ago
  I know, my point is that when training demand decreases people will be realize that inference does not make up the difference
  - nextworddev 2y ago
    Yeah the big question I’m struggling with is exactly when training demand will fall if at all
    - sroussey 2y ago
      Every research lab is focused on new architectures that would reduce training costs.
      - nextworddev 2y ago
        Yeah we need essentially hadoop for llm training
    - vineyardmike 2y ago
      Well every other tech company is writing checks to prove they have the chops to make an LLM. From IBM to Databricks to the big guys like Google. Tons of companies made one just to show off to investors or their CEO or just because they wanted skin in the game. But that probably won't continue forever. We've already seen certain orgs that seem to outshine others, and if they can't catch up, they may just accept to using the open-access models or APIs instead. At some point everyone will realize that it is becoming a commodity, and it is very expensive to train, then only those with wither a structural advantage to lower price (eg Google) or a true goal of being on the high-end/SOTA of the market (OpenAI, Anthropic) will keep going.
      - fauigerzigerk 2y ago
        You're talking about training the foundation models. But what about all the fine tuning on non-public business data that will be necessary to make gen AI useful for actual business processes? I'm finding it difficult to estimate the size of this workload compared to continued training of foundation models. Perhaps it depends on whether there are new architectural breakthroughs that require retraining of foundation models. And what about non-language tasks such as interpreting video and 3D sensory data? This is potentially huge, but between huge peaks there is often a valley of unknowable depth and breadth.
- amluto 2y ago
  Nvidia doesn’t obviously have a strong inference play right now for a widely-deployed small model. For a model that really needs a 4090, maybe. But for a model that can run on a Coral chip or an M1/M2/M3 or whatever Intel or AMD’s latest little AI engines can do? This market has plenty of players, and Nvidia doesn’t seem to be anywhere near the lead except insofar as it’s a little bit easier to run the software on CUDA.
smokel 2y ago
As someone else points out, training is slightly more involved, but I also find that these smaller models are next to worthless compared to the larger ones. There are probably some situations where it suffices to use a small model, but for most purposes, I'd prefer to use the state of the art, and I'm eager for that state to progress a little more.
- vineyardmike 2y ago
  > but for most purposes, I'd prefer to use the state of the art I'm guessing in the future, we'll see a lot more automatic inference "on our behalf", and you won't care or notice if its using Llama5:3b or whatever comes out then. I'm betting that in a few years we'll see LLMs baked into a ton of stuff - lots of "simple" things mostly, like email summary/rewording, and similar "light touch" use cases. Maybe light multi-modal work like photo labeling. Probably expanded to be running in every other SaaS applications doing god knows what. For those, the small models would be more than enough, and much cheaper to run in large volumes. I'm guessing "chat with a bot to ask questions" will be a small amount of the inference that happens on our behalf, but will use the valuable SOTA model use case.
dwaltrip 2y ago
> LLM performance is plateauing It’s a wee bit early to call this. Let’s see what the top labs release in the next year or two, yeah? GPT-4 was released only 15 months ago, which was about 3 years after GPT-3 was released. These things don’t happen overnight, and many multi-year efforts are currently in the works, especially starting last year.
riku_iki 2y ago
> I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing next wave driving demand can be actual new products developed on LLMs. There are very few usecases currently well developed besides chatbots, but potential is very large.

epolanski 2y ago

I am a partial believer that the real race for many tech players is actually AGI and ASI later and till the problem is solved the hardware arm race will keep being part of it. Not only big tech is part of it but billion dollars startups are popping everywhere from China to US and Middle East.

chrisdbanks 2y ago
Unless we hit another AI winter. We might get to the point where the hardware just can't give better returns and have to wait another 20 years for the next leap forward. We're still orders of magnitude away from the human brain.

deleted 2y ago

[deleted]

nextworddev 2y ago

It’s actually about training, not inference. You can’t do training on commodity gpus but yeah once someone figures that out, nvdia could crash

gradus_ad 2y ago
I know, my point is that when training demand decreases people will be realize that inference does not make up the difference
- nextworddev 2y ago
  Yeah the big question I’m struggling with is exactly when training demand will fall if at all
  - sroussey 2y ago
    Every research lab is focused on new architectures that would reduce training costs.
    - nextworddev 2y ago
      Yeah we need essentially hadoop for llm training
  - vineyardmike 2y ago
    Well every other tech company is writing checks to prove they have the chops to make an LLM. From IBM to Databricks to the big guys like Google. Tons of companies made one just to show off to investors or their CEO or just because they wanted skin in the game. But that probably won't continue forever. We've already seen certain orgs that seem to outshine others, and if they can't catch up, they may just accept to using the open-access models or APIs instead. At some point everyone will realize that it is becoming a commodity, and it is very expensive to train, then only those with wither a structural advantage to lower price (eg Google) or a true goal of being on the high-end/SOTA of the market (OpenAI, Anthropic) will keep going.
    - fauigerzigerk 2y ago
      You're talking about training the foundation models. But what about all the fine tuning on non-public business data that will be necessary to make gen AI useful for actual business processes? I'm finding it difficult to estimate the size of this workload compared to continued training of foundation models. Perhaps it depends on whether there are new architectural breakthroughs that require retraining of foundation models. And what about non-language tasks such as interpreting video and 3D sensory data? This is potentially huge, but between huge peaks there is often a valley of unknowable depth and breadth.
amluto 2y ago
Nvidia doesn’t obviously have a strong inference play right now for a widely-deployed small model. For a model that really needs a 4090, maybe. But for a model that can run on a Coral chip or an M1/M2/M3 or whatever Intel or AMD’s latest little AI engines can do? This market has plenty of players, and Nvidia doesn’t seem to be anywhere near the lead except insofar as it’s a little bit easier to run the software on CUDA.

smokel 2y ago

As someone else points out, training is slightly more involved, but I also find that these smaller models are next to worthless compared to the larger ones. There are probably some situations where it suffices to use a small model, but for most purposes, I'd prefer to use the state of the art, and I'm eager for that state to progress a little more.

vineyardmike 2y ago
> but for most purposes, I'd prefer to use the state of the art I'm guessing in the future, we'll see a lot more automatic inference "on our behalf", and you won't care or notice if its using Llama5:3b or whatever comes out then. I'm betting that in a few years we'll see LLMs baked into a ton of stuff - lots of "simple" things mostly, like email summary/rewording, and similar "light touch" use cases. Maybe light multi-modal work like photo labeling. Probably expanded to be running in every other SaaS applications doing god knows what. For those, the small models would be more than enough, and much cheaper to run in large volumes. I'm guessing "chat with a bot to ask questions" will be a small amount of the inference that happens on our behalf, but will use the valuable SOTA model use case.

dwaltrip 2y ago

> LLM performance is plateauing It’s a wee bit early to call this. Let’s see what the top labs release in the next year or two, yeah? GPT-4 was released only 15 months ago, which was about 3 years after GPT-3 was released. These things don’t happen overnight, and many multi-year efforts are currently in the works, especially starting last year.

riku_iki 2y ago

> I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing next wave driving demand can be actual new products developed on LLMs. There are very few usecases currently well developed besides chatbots, but potential is very large.