6 ms·

I agree I don't understand the difference? Are the calculations an NPU is capable of doing different to a GPU? Are they not basically identical hardware?

klelatti 2y ago
NPUs are basically specialized for matrix multiplication, GPUs for more general parallel operations on multiple data (Single Instruction Multiple Thread) although modern GPUs also contain matrix multiplication units. May be a degree of software compatibility at the highest level - eg PyTorch - but the underlying software will be very different.
adrian_b 2y ago
A NPU can do only a very small subset of the operations supported by a GPU. A NPU does strictly only the operations required for ML inference, which use data types with low precision, i.e. 16-bit or 8-bit types.
foobiekr 2y ago
Different optimization choices.
xcv123 2y ago
By the same reasoning, a CPU is no different to a GPU. They can both do matrix calculations. A GPU is optimised for 3D rendering (and is useful for parallel computations in general). An NPU is optimised for neural network inferencing. These algorithms both involve matrix mathematics but they are not the same. The NPU hardware design matches the deep neural network inferencing algorithm. For example it has an "Activation Function" block dedicated to computing the activation function between neural network layers. It is optimised and specialised for one very specific algorithm: inferencing. A GPU would beat an NPU for training, and any other parallel computations besides inferencing.
- TheDudeMan 2y ago
  High-end Nvidia GPUs are not optimized for 3D rendering; they are optimized for machine learning and inference.
  - taneq 2y ago
    They're not really GPUs then, are they? Even if they're capable of generating a video output.
    - Dalewyn 2y ago
      They are GPUs, but standing for General Processing Unit. "GPU" the acronym has persisted from sheer force of tradition and habit, but what it means has changed drastically over the past several years between cryptocurrency and now "AI". I wonder if NPU will supercede GPU as in General Processing Unit now that it has finally entered the wider lexicon, relegating GPU back to Graphics Processing Unit or video cards. And no, GPGPU (General Purpose Graphics Processing Unit) is a bloody stupid term to be bluntly honest.
      - taneq 2y ago
        That might be how you use the term but GPU is still almost universally used to mean graphics processing unit. If it want to use a ubiquitous acronym to mean something different, you need to define it first.
        Dalewyn 2y ago
        >GPU is still almost universally used to mean graphics processing unit. No, GPU is almost universally used to mean GPU. There is nothing graphical about cryptocurrency, "AI" (sans image generation), protein crunching, and whatever else they are being used for that aren't graphical. I question how many people are even aware the G is supposed to stand for Graphics anymore. The nomenclature is outdated and doesn't reflect reality anymore.
        smcin 2y ago
- int_19h 2y ago
  Modern GPUs have cores optimized for different things, including specifically things that NPUs (as defined by the article) are all about. E.g. here's NVidia: https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/ https://developer.nvidia.com/blog/programming-tensor-cores-c...
Const-me 2y ago
> Are they not basically identical hardware? For example Apple’s NPU can’t do FP32 precision, it can only do FP16 and less.

klelatti 2y ago

NPUs are basically specialized for matrix multiplication, GPUs for more general parallel operations on multiple data (Single Instruction Multiple Thread) although modern GPUs also contain matrix multiplication units. May be a degree of software compatibility at the highest level - eg PyTorch - but the underlying software will be very different.

adrian_b 2y ago

A NPU can do only a very small subset of the operations supported by a GPU. A NPU does strictly only the operations required for ML inference, which use data types with low precision, i.e. 16-bit or 8-bit types.

foobiekr 2y ago

Different optimization choices.

xcv123 2y ago

By the same reasoning, a CPU is no different to a GPU. They can both do matrix calculations. A GPU is optimised for 3D rendering (and is useful for parallel computations in general). An NPU is optimised for neural network inferencing. These algorithms both involve matrix mathematics but they are not the same. The NPU hardware design matches the deep neural network inferencing algorithm. For example it has an "Activation Function" block dedicated to computing the activation function between neural network layers. It is optimised and specialised for one very specific algorithm: inferencing. A GPU would beat an NPU for training, and any other parallel computations besides inferencing.

TheDudeMan 2y ago
High-end Nvidia GPUs are not optimized for 3D rendering; they are optimized for machine learning and inference.
- taneq 2y ago
  They're not really GPUs then, are they? Even if they're capable of generating a video output.
  - Dalewyn 2y ago
    They are GPUs, but standing for General Processing Unit. "GPU" the acronym has persisted from sheer force of tradition and habit, but what it means has changed drastically over the past several years between cryptocurrency and now "AI". I wonder if NPU will supercede GPU as in General Processing Unit now that it has finally entered the wider lexicon, relegating GPU back to Graphics Processing Unit or video cards. And no, GPGPU (General Purpose Graphics Processing Unit) is a bloody stupid term to be bluntly honest.
    - taneq 2y ago
      That might be how you use the term but GPU is still almost universally used to mean graphics processing unit. If it want to use a ubiquitous acronym to mean something different, you need to define it first.
      - Dalewyn 2y ago
        >GPU is still almost universally used to mean graphics processing unit. No, GPU is almost universally used to mean GPU. There is nothing graphical about cryptocurrency, "AI" (sans image generation), protein crunching, and whatever else they are being used for that aren't graphical. I question how many people are even aware the G is supposed to stand for Graphics anymore. The nomenclature is outdated and doesn't reflect reality anymore.
        smcin 2y ago
int_19h 2y ago
Modern GPUs have cores optimized for different things, including specifically things that NPUs (as defined by the article) are all about. E.g. here's NVidia: https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/ https://developer.nvidia.com/blog/programming-tensor-cores-c...

Const-me 2y ago

> Are they not basically identical hardware? For example Apple’s NPU can’t do FP32 precision, it can only do FP16 and less.