3 ms·

Isn't the current GPU-based stack which drives the progress of AI an early stage architecture which will become sub-optimal and obsolete in a longer term? Havi

Isn't the current GPU-based stack which drives the progress of AI an early stage architecture which will become sub-optimal and obsolete in a longer term? Having separate processors with separate memory and separate software stack to do matrix operations works, but it would be much more convenient and productive to have a system with one RAM and CPUs that can efficiently do matrix operations and would not require the programmer to delegate these operations to a separate stack. Event the name 'Graphics Processing Unit' suggest that the current approach for AI is rather hacky. Because of this, in a long run there can an opportunity for Intel and other CPU manufacturers to regain the lucrative market from NVIDIA.

mike_hearn 3y ago
The software stack co-evolves with the hardware, if the hardware can't do something fast then the software guys can't necessarily even try it out. People have been trying to break NVIDIA's moat by creating non-GPU accelerators. The core maths operations aren't that varied so it's an obvious thing to try. Nothing worked, not even NVIDIA's own attempts. AI researchers have a software stack oriented around devices with separate memory and it's abstracted, so unifying CPU and GPU ram doesn't seem to make a big difference for them. Don't be fooled by the name GPU. It's purely historical. "GPUs" used for AI don't have any video output capability. I think you can't even render with them. They are just specialised computers come with their own OS ("driver"), compilers, APIs and dev stack, which happen to need a CPU to bring them online and which communicate via PCIe instead of ethernet.
- drBonkers 3y ago
  How do the new “TPUs” and “NPUs” slot into your perspective of GPUs? Just nomenclature catching up to describe matrix math, ML GPUs without video output?
  - mike_hearn 3y ago
    TPU is a Google thing. It shows how far NVIDIA has gone from classical GPUs that Google made ML specific silicon and it's not a clear winner over NVIDIA chips (or in some cases is a clear loser, from what I understand). I don't know about NPUs. Do you mean Apple? Apple silicon is interesting because of the unified memory architecture, but beyond letting you save RAM by using it for both graphics and app code simultaneously I'm not sure it has much benefit. Certainly, datacenters will remain on NVIDIA for the time being unless AMD manage to catch up on their software stack. Intel is the dark horse here. I hear rumblings that their first offering is actually quite good.
goriloser 3y ago
Fast memory is very expensive. Which is why you only use it when you really need it - on a GPU/AI accelerator.

mike_hearn 3y ago

The software stack co-evolves with the hardware, if the hardware can't do something fast then the software guys can't necessarily even try it out. People have been trying to break NVIDIA's moat by creating non-GPU accelerators. The core maths operations aren't that varied so it's an obvious thing to try. Nothing worked, not even NVIDIA's own attempts. AI researchers have a software stack oriented around devices with separate memory and it's abstracted, so unifying CPU and GPU ram doesn't seem to make a big difference for them. Don't be fooled by the name GPU. It's purely historical. "GPUs" used for AI don't have any video output capability. I think you can't even render with them. They are just specialised computers come with their own OS ("driver"), compilers, APIs and dev stack, which happen to need a CPU to bring them online and which communicate via PCIe instead of ethernet.

drBonkers 3y ago
How do the new “TPUs” and “NPUs” slot into your perspective of GPUs? Just nomenclature catching up to describe matrix math, ML GPUs without video output?
- mike_hearn 3y ago
  TPU is a Google thing. It shows how far NVIDIA has gone from classical GPUs that Google made ML specific silicon and it's not a clear winner over NVIDIA chips (or in some cases is a clear loser, from what I understand). I don't know about NPUs. Do you mean Apple? Apple silicon is interesting because of the unified memory architecture, but beyond letting you save RAM by using it for both graphics and app code simultaneously I'm not sure it has much benefit. Certainly, datacenters will remain on NVIDIA for the time being unless AMD manage to catch up on their software stack. Intel is the dark horse here. I hear rumblings that their first offering is actually quite good.

goriloser 3y ago

Fast memory is very expensive. Which is why you only use it when you really need it - on a GPU/AI accelerator.