4 ms·

The article says Microsoft wants computers to one day have Neural Processing Units (NPU) like most have GPUs. If that led to faster suggestions for copilot, I'd

The article says Microsoft wants computers to one day have Neural Processing Units (NPU) like most have GPUs. If that led to faster suggestions for copilot, I'd be all for it. Right now, a GPU is an NPU. Does anyone how an NPU would differ from a graphics card?

dragontamer 3y ago
GPUs are a SIMD architecture designed for 4x4 float32 matrix multiplications found in video games. But NPUs (like Googles NPU) are systolic arrays designed for 16x16 or even 256x256 float16 or even int8 matrix multiplications instead. ------- NVidia builds larger matrix multiplications out of the 4x4 float16 base that a SM is designed for. After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications.
- dmurray 3y ago
  > After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications Isn't it 8 of them? That's why you can't go the other way around and use your 8x8 multiplier to efficiently multiply 4x4s in parallel.
  - uxp8u61q 3y ago
    Yes, it's eight multiplication with the naive algorithm. You can get to seven with a clever trick (Strassen's algorithm). As far as I'm aware, with these precise numbers (8x8 matrix decomposed into four 4x4) it's unknown if we can do better.
  - dragontamer 3y ago
    Yes, it is 8. I was mistaken earlier.
Squarex 3y ago
isn't it what neural engine on apple silicon is?
- hospitalJail 3y ago
  Integrated vs Discrete.
0x008 3y ago
I think Meta (facebook) is already working with android phone manufacturers to include chips with specific instruction sets to support the llama architecture. probably intel is working on the same. I assume the next generation of mobile chips (laptop or phone) will all include special LLM processors.
WithinReason 3y ago
If you already have a GPU, adding tensor cores to it is more optimal (less area=cheaper) than adding a separate NPU.
visarga 3y ago
> Does anyone how an NPU would differ from a graphics card? Compare NVIDIA to Groq TSP (Tensor Streaming Processor) This is how an AI chip should be designed, you write the neural net compiler first, then design the chip. These guys have combined a number of smart ideas in their product: synchronous operation makes memory and network operations simpler to plan and optimise; all software defined memory access, no caches; a simple set of primitive operations that can express every model architecture - so they can simply make it pytorch compatible, no need to write kernels, no need to have 100 versions of 2D conv, for all sizes and shapes. https://groq.com/groq-tensor-streaming-processor-architecture-is-radically-different/ https://groq.com/groq-tensor-streaming-processor-architectur... Groq recently demonstrated 240T/s for LLaMA-2 70B. Probably world record, the next best I know is 40T/s at labs.perplexity.ai https://youtu.be/hk4QGpQAvSY?t=57 https://youtu.be/hk4QGpQAvSY?t=57
- bloopernova 3y ago
  Very cool stuff, thank you for the new information

dragontamer 3y ago

GPUs are a SIMD architecture designed for 4x4 float32 matrix multiplications found in video games. But NPUs (like Googles NPU) are systolic arrays designed for 16x16 or even 256x256 float16 or even int8 matrix multiplications instead. ------- NVidia builds larger matrix multiplications out of the 4x4 float16 base that a SM is designed for. After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications.

dmurray 3y ago
> After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications Isn't it 8 of them? That's why you can't go the other way around and use your 8x8 multiplier to efficiently multiply 4x4s in parallel.
- uxp8u61q 3y ago
  Yes, it's eight multiplication with the naive algorithm. You can get to seven with a clever trick (Strassen's algorithm). As far as I'm aware, with these precise numbers (8x8 matrix decomposed into four 4x4) it's unknown if we can do better.
- dragontamer 3y ago
  Yes, it is 8. I was mistaken earlier.

Squarex 3y ago

isn't it what neural engine on apple silicon is?

hospitalJail 3y ago
Integrated vs Discrete.

0x008 3y ago

I think Meta (facebook) is already working with android phone manufacturers to include chips with specific instruction sets to support the llama architecture. probably intel is working on the same. I assume the next generation of mobile chips (laptop or phone) will all include special LLM processors.

WithinReason 3y ago

If you already have a GPU, adding tensor cores to it is more optimal (less area=cheaper) than adding a separate NPU.

visarga 3y ago

> Does anyone how an NPU would differ from a graphics card? Compare NVIDIA to Groq TSP (Tensor Streaming Processor) This is how an AI chip should be designed, you write the neural net compiler first, then design the chip. These guys have combined a number of smart ideas in their product: synchronous operation makes memory and network operations simpler to plan and optimise; all software defined memory access, no caches; a simple set of primitive operations that can express every model architecture - so they can simply make it pytorch compatible, no need to write kernels, no need to have 100 versions of 2D conv, for all sizes and shapes. https://groq.com/groq-tensor-streaming-processor-architecture-is-radically-different/ https://groq.com/groq-tensor-streaming-processor-architectur... Groq recently demonstrated 240T/s for LLaMA-2 70B. Probably world record, the next best I know is 40T/s at labs.perplexity.ai https://youtu.be/hk4QGpQAvSY?t=57 https://youtu.be/hk4QGpQAvSY?t=57

bloopernova 3y ago
Very cool stuff, thank you for the new information