3 ms·

> is possibly a hardware dead-end. I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventu

> is possibly a hardware dead-end. I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. Just by seeing how morbid are the sales of the new "NPUs," I can guess it's already happening. A number of cellphone brands experimented with such to run selfie filters, or do speech recognition, but later found that those work on CPUs not any worse at all if competent programmers are hired, and then threw the NPU hardware out, or disabled it.

dragontamer 6y ago
> I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. There's only been two styles: * SIMD style compute (which has been around since Cray and CM-2, and continues today with AMD GPUs and NVidia GPUs). * Systolic-style compute -- which has been around since CD-ROM ASICs for decoding the CD-ROM realtime as it was spinning. Apparently Reed-Solomon decoding needed a ton of matrix-multiplications. Modern "NPU Hardware" is simply systolic compute. ------------ Traditional CPUs load data and then store data. The core itself can remain unchanging in the demands of different applications. Systolic Compute always loads/stores data in the same order, as it traverses across your systolic array. The systolic array designed for CD-ROM playback is quite different than the systolic array for NPUs. You gain faster speeds by building the hardware itself to handle the load/store pattern of the algorithm. That way the CPU-core itself doesn't waste time on loads/stores, nor do you have to worry about memory bandwidth issues or whatever. But by doing so: you lock yourself into a particular application. Modern neural-net NPUs can only do FP16 matrix multiplications. That's fine. If you want the fastest neural net in the world, systolic arrays will get you there. But that thing will ONLY be able to do neural net calculations, nothing else. Maybe you build the systolic array into an FPGA so that you can reconfigure the hardware later. Or maybe you just go with an ASIC because you're Google and can afford to build out clusters of ASICs of Systolic Arrays that can only do tensor-calculations.
- touisteur 6y ago
  When you look at how mind-bendingly precise surrogate NNs can be, sometimes you build your pipeline with 'slow' FP16 NNs and it's still faster than the 'direct' operation. I also like how using tensor cores at the same time as cuda cores is how you get 'more' performance. It's a bit like CPU execution ports, if you /don't/ use them you're leaving performance on the floor. Only here nvidia (cuda, tensor, rt) 'cores' are a bit higher level than exec ports. Also it's kind of hard (i.e. fun) to find a use for rt cores. Digging up all those 90's papers about exotic applications of Raytracing (I mean weed must have been great then :-) Interesting times.

dragontamer 6y ago

> I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. There's only been two styles: * SIMD style compute (which has been around since Cray and CM-2, and continues today with AMD GPUs and NVidia GPUs). * Systolic-style compute -- which has been around since CD-ROM ASICs for decoding the CD-ROM realtime as it was spinning. Apparently Reed-Solomon decoding needed a ton of matrix-multiplications. Modern "NPU Hardware" is simply systolic compute. ------------ Traditional CPUs load data and then store data. The core itself can remain unchanging in the demands of different applications. Systolic Compute always loads/stores data in the same order, as it traverses across your systolic array. The systolic array designed for CD-ROM playback is quite different than the systolic array for NPUs. You gain faster speeds by building the hardware itself to handle the load/store pattern of the algorithm. That way the CPU-core itself doesn't waste time on loads/stores, nor do you have to worry about memory bandwidth issues or whatever. But by doing so: you lock yourself into a particular application. Modern neural-net NPUs can only do FP16 matrix multiplications. That's fine. If you want the fastest neural net in the world, systolic arrays will get you there. But that thing will ONLY be able to do neural net calculations, nothing else. Maybe you build the systolic array into an FPGA so that you can reconfigure the hardware later. Or maybe you just go with an ASIC because you're Google and can afford to build out clusters of ASICs of Systolic Arrays that can only do tensor-calculations.

touisteur 6y ago
When you look at how mind-bendingly precise surrogate NNs can be, sometimes you build your pipeline with 'slow' FP16 NNs and it's still faster than the 'direct' operation. I also like how using tensor cores at the same time as cuda cores is how you get 'more' performance. It's a bit like CPU execution ports, if you /don't/ use them you're leaving performance on the floor. Only here nvidia (cuda, tensor, rt) 'cores' are a bit higher level than exec ports. Also it's kind of hard (i.e. fun) to find a use for rt cores. Digging up all those 90's papers about exotic applications of Raytracing (I mean weed must have been great then :-) Interesting times.