5 ms·

If AI remains in the cloud, nvidia wins. But I can’t help but think that if AI becomes “self-hosted”, if we return to a world where people own their own machine

If AI remains in the cloud, nvidia wins. But I can’t help but think that if AI becomes “self-hosted”, if we return to a world where people own their own machines, AMDs APUs and interconnect technology will be absolutely dominant. Training may still be Nvidias wheelhouse, but for a single device able to do all the things (inference, rendering, and computing), AMD, at least currently, would seem to be the winner. I’d love someone more knowledgeable in AI scaling to correct me here though. Maybe that’s all far enough afield to make the current state of things irrelevant?

teaearlgraycold 2y ago
You need orders of magnitude more compute for training than for inference. Nvidia still wins in your scenario. Currently rendering and local GPGPU compute is Nvidia dominated and I don’t see AMD competently going after the market segments.
- demaga 2y ago
  But you also run inference orders of magnitudes more times, so it should still amount to more compute than training?
  - teaearlgraycold 2y ago
    That matters more to the electricity company than the silicon company. The profit margins on the datacenter training hardware are stupidly high compared to an AMD APU.
    - binary132 2y ago
      If there are tens of thousands of training GPUs but billions of APUs, then what? BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there.
      - k__ 2y ago
        This. Most will probably use something like Llama as base.
      - talldayo 2y ago
        > If there are tens of thousands of training GPUs but billions of APUs, then what? Believe it or not, we've actually been grappling with this scenario for almost a decade at this point. Originally the answer was to unite hardware manufacturers around a common featureset that could compete with (albeit not replace) CUDA. Khronos was prepared to elevate OpenCL to an industry standard, but Apple pulled their support for it and let the industry collapse into proprietary competition again. I bet they're kicking themselves over that one, if they still hold a stronger grudge against Nvidia than Khronos at least. So - logically, there's actually a one-size-fits-all solution for this problem. It was even going to get managed by the same people handling Vulkan. The problem was corporate greed and shortsighted investment that let OpenCL languish while CUDA was under active heavy development. > BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there. Eh, that's kinda like saying "app development is so expensive that consumers will eventually care". Consumers just buy the end product; they are never exposed to building the software or concerned with the cost of the development. This is especially true with businesses like OpenAI that just give you free access to a decent LLM (or Apple and their "it's free for now" mentality).
  - marcosdumay 2y ago
    Besides, if you separate them, the people doing the training will put way more effort into optimizing their hardware ROI than the ones doing inference.
- mistercow 2y ago
  I think this is the big point of uncertainty in Nvidia’s future: will we find new training techniques which require significantly less compute, and/or are better suited to some fundamentally different architecture than GPUs? I’m reluctant to bet no on that long term, and “long term” for ML right now is not very long.
  - brigadier132 2y ago
    If we find a new training technique that is that much more efficient why do you think we wont just increase the amount of training we do be n times? (or even more since it's now accessible to train custom models for smaller businesses)
    - mistercow 2y ago
      We might, but it’s also plausible that it would change the ecosystem so much that centralized models are no longer so prominent. For example, suppose that with much cheaper training, most training is on your specific data and behaviors so that you have a model (or ensemble of models) tailored to your own needs. You still need a foundation model, but those are much smaller so that they can run on device, so even with overparameterization and distillation, the training costs are orders of magnitude smaller. Or, in the small business case (mind you, “long term” for tech reaching small businesses is looooong), these businesses again need much smaller models because a) they don’t need a model well versed in Shakespeare and multi variable calculus, and b) they want inference to be as low cost as possible. These are just scenarios off the top of my head. The broader point is that a dramatic drop in training cost is a wildcard whose effects are really hard to predict.
  - marcosdumay 2y ago
    I'd bet that any AI that is really useful for the tasks people want to push LLMs into will answer "yes" to both parts of your question. But I don't know what "long term" is exactly, and have no idea how to time this thing. Besides, I'd bet the sibling evoking the Jevon's paradox is correct.
  - acchow 2y ago
passion__desire 2y ago
If compute is gonna play the role of electricity in coming decades, then having a compute wall similar to Tesla powerwall is a necessity.
- ta988 2y ago
  Only if improvements in speed and energy savings slow down
  - CooCooCaCha 2y ago
    And if models don't get any larger, which they will
- CooCooCaCha 2y ago
  Powerwall and electric car in the garage, compute wall in the closet, 3d printer and other building tools in the manufacturing room, hydroponics setup in the indoor farm room, and AI assistant to help manage it all. The home becomes a mixed living area and factory.
  - crowcroft 2y ago
    The vision of this sounds so cool, but man, for a lot of use cases at the moment most 'smart home' stuff is still complicated and temperamental. How do we get from here to there, cause I want to get there so bad.
    - CooCooCaCha 2y ago
      Tech and economic development can help a bit, like the new Bamboo 3d printer makes 3d printing a lot more "idiot proof". However, I think we need AI beyond current LLMs to really take us there. I'm not saying LLMs can't get us there, we don't know, just beyond what we have. We need AI that we can trust with real tasks IRL.
- rbanffy 2y ago
  Powerwall makes sense because you can’t generate energy at any time and, therefore, you store it. Computers are not like that - you don’t “store” computations for when you need them - you either use capacity or you don’t. That makes it practical to centralise computing and only pay for what you use.
moffkalast 2y ago
Nvidia still has 12-16GB VRAM offerings for around $300-400, which are exceptionally well optimized and supported on the software side. Still by far the most cost effective option if you also value your time imo. The Strix Halo better have high tier Mac level bandwidth plus ROCm support and be priced below $1k or it's just not competitive with that because it'll still be slower than even partial cuda offloading.
- halJordan 2y ago
  It has 80 GBps bw from its dual channel ddr5 implementations. I really dont think AMD is dogfooding any of their ML toolkits, which is a shame.
wmf 2y ago
You may be seeing something that isn't there. I don't even know if MI300A is available to buy, what it costs, or if you'll be forced to buy four of them which would push prices close to DGX territory anyway.
BaculumMeumEst 2y ago
I remember reading geohot advocate for 7900XTX as a cost effective card for deep learning. I read AMD is backing off from the high end GPU market, though. Is there any chance they will at least continue to offer cards with lots of VRAM?
acchow 2y ago
The cloud is more efficient at utilizing hardware. Except for low-latency or low-connection requirements, the move to cloud will continue.
- aurareturn 2y ago
  No, there will be plenty of low value inference that won't be economical in the cloud. Apple Intelligence is one example.

teaearlgraycold 2y ago

You need orders of magnitude more compute for training than for inference. Nvidia still wins in your scenario. Currently rendering and local GPGPU compute is Nvidia dominated and I don’t see AMD competently going after the market segments.

demaga 2y ago
But you also run inference orders of magnitudes more times, so it should still amount to more compute than training?
- teaearlgraycold 2y ago
  That matters more to the electricity company than the silicon company. The profit margins on the datacenter training hardware are stupidly high compared to an AMD APU.
  - binary132 2y ago
    If there are tens of thousands of training GPUs but billions of APUs, then what? BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there.
    - k__ 2y ago
      This. Most will probably use something like Llama as base.
    - talldayo 2y ago
      > If there are tens of thousands of training GPUs but billions of APUs, then what? Believe it or not, we've actually been grappling with this scenario for almost a decade at this point. Originally the answer was to unite hardware manufacturers around a common featureset that could compete with (albeit not replace) CUDA. Khronos was prepared to elevate OpenCL to an industry standard, but Apple pulled their support for it and let the industry collapse into proprietary competition again. I bet they're kicking themselves over that one, if they still hold a stronger grudge against Nvidia than Khronos at least. So - logically, there's actually a one-size-fits-all solution for this problem. It was even going to get managed by the same people handling Vulkan. The problem was corporate greed and shortsighted investment that let OpenCL languish while CUDA was under active heavy development. > BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there. Eh, that's kinda like saying "app development is so expensive that consumers will eventually care". Consumers just buy the end product; they are never exposed to building the software or concerned with the cost of the development. This is especially true with businesses like OpenAI that just give you free access to a decent LLM (or Apple and their "it's free for now" mentality).
- marcosdumay 2y ago
  Besides, if you separate them, the people doing the training will put way more effort into optimizing their hardware ROI than the ones doing inference.
mistercow 2y ago
I think this is the big point of uncertainty in Nvidia’s future: will we find new training techniques which require significantly less compute, and/or are better suited to some fundamentally different architecture than GPUs? I’m reluctant to bet no on that long term, and “long term” for ML right now is not very long.
- brigadier132 2y ago
  If we find a new training technique that is that much more efficient why do you think we wont just increase the amount of training we do be n times? (or even more since it's now accessible to train custom models for smaller businesses)
  - mistercow 2y ago
    We might, but it’s also plausible that it would change the ecosystem so much that centralized models are no longer so prominent. For example, suppose that with much cheaper training, most training is on your specific data and behaviors so that you have a model (or ensemble of models) tailored to your own needs. You still need a foundation model, but those are much smaller so that they can run on device, so even with overparameterization and distillation, the training costs are orders of magnitude smaller. Or, in the small business case (mind you, “long term” for tech reaching small businesses is looooong), these businesses again need much smaller models because a) they don’t need a model well versed in Shakespeare and multi variable calculus, and b) they want inference to be as low cost as possible. These are just scenarios off the top of my head. The broader point is that a dramatic drop in training cost is a wildcard whose effects are really hard to predict.
- marcosdumay 2y ago
  I'd bet that any AI that is really useful for the tasks people want to push LLMs into will answer "yes" to both parts of your question. But I don't know what "long term" is exactly, and have no idea how to time this thing. Besides, I'd bet the sibling evoking the Jevon's paradox is correct.
- acchow 2y ago

passion__desire 2y ago

If compute is gonna play the role of electricity in coming decades, then having a compute wall similar to Tesla powerwall is a necessity.

ta988 2y ago
Only if improvements in speed and energy savings slow down
- CooCooCaCha 2y ago
  And if models don't get any larger, which they will
CooCooCaCha 2y ago
Powerwall and electric car in the garage, compute wall in the closet, 3d printer and other building tools in the manufacturing room, hydroponics setup in the indoor farm room, and AI assistant to help manage it all. The home becomes a mixed living area and factory.
- crowcroft 2y ago
  The vision of this sounds so cool, but man, for a lot of use cases at the moment most 'smart home' stuff is still complicated and temperamental. How do we get from here to there, cause I want to get there so bad.
  - CooCooCaCha 2y ago
    Tech and economic development can help a bit, like the new Bamboo 3d printer makes 3d printing a lot more "idiot proof". However, I think we need AI beyond current LLMs to really take us there. I'm not saying LLMs can't get us there, we don't know, just beyond what we have. We need AI that we can trust with real tasks IRL.
rbanffy 2y ago
Powerwall makes sense because you can’t generate energy at any time and, therefore, you store it. Computers are not like that - you don’t “store” computations for when you need them - you either use capacity or you don’t. That makes it practical to centralise computing and only pay for what you use.

moffkalast 2y ago

Nvidia still has 12-16GB VRAM offerings for around $300-400, which are exceptionally well optimized and supported on the software side. Still by far the most cost effective option if you also value your time imo. The Strix Halo better have high tier Mac level bandwidth plus ROCm support and be priced below $1k or it's just not competitive with that because it'll still be slower than even partial cuda offloading.

halJordan 2y ago
It has 80 GBps bw from its dual channel ddr5 implementations. I really dont think AMD is dogfooding any of their ML toolkits, which is a shame.

wmf 2y ago

You may be seeing something that isn't there. I don't even know if MI300A is available to buy, what it costs, or if you'll be forced to buy four of them which would push prices close to DGX territory anyway.

BaculumMeumEst 2y ago

I remember reading geohot advocate for 7900XTX as a cost effective card for deep learning. I read AMD is backing off from the high end GPU market, though. Is there any chance they will at least continue to offer cards with lots of VRAM?

acchow 2y ago

The cloud is more efficient at utilizing hardware. Except for low-latency or low-connection requirements, the move to cloud will continue.

aurareturn 2y ago
No, there will be plenty of low value inference that won't be economical in the cloud. Apple Intelligence is one example.