4 ms·

I wonder why we don't have external "neural processing" devices like we once had soundcards. Is anyone working on hardware implementation of transformers? Kudo

by seydor 4y ago

I wonder why we don't have external "neural processing" devices like we once had soundcards. Is anyone working on hardware implementation of transformers? Kudos to Yann lecun for getting his revenge for Galactica

jhrmnn 4y ago
https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- seydor 4y ago
  but those are not for sale, and not transformer-specific. There must be some optimizations that can be done in hardware and transformers are several years old now
  - ruuda 4y ago
    You likely already bought one. https://blog.google/products/pixel/introducing-google-tensor/ https://blog.google/products/pixel/introducing-google-tensor... https://apple.fandom.com/wiki/Neural_Engine https://apple.fandom.com/wiki/Neural_Engine
  - jhrmnn 4y ago
    Computation-wise, transformers are really just a bunch of matrix multiplications, nothing more to it. (Which is partially why they’re so efficient and scalable.) Also, Nvidia’s GPU architectures are moving in the TPU direction (https://www.nvidia.com/en-us/data-center/tensor-cores/ https://www.nvidia.com/en-us/data-center/tensor-cores/).
    - shaklee3 4y ago
      Nvidia had tensor cores before the tpu existed
      - jhrmnn 4y ago
        Right, correct. I was already thinking ahead about the recent TensorFloat precision, which is further directly targeting matrix multiplication.
      - dragonwriter 4y ago
        > Nvidia had tensor cores before the tpu existed TPUs were released in 2016, Tensor Cores in 2017. But Edge TPUs (that is, the ones that exist outside of Google's Cloud infrastructure) weren’t until 2018. So, Nvidia had their cores available for people to buy first, but not before TPUs existed.
  - dragonwriter 4y ago
    Google TPU are for sale [0] as are Nvidia cards with their Tensor Cores. As are Apple mainboards with their Neural Engine. [0] https://coral.ai/products/ https://coral.ai/products/
zenogantner 4y ago
> wonder why we don't have external "neural processing" devices like we once had soundcards. Some video cards/GPUs have become just that, becoming more and more geared towards non-graphics workloads ...
valine 4y ago
Nvidia A100 is exactly that. It has lower cuda performance than a RTX 4090, and is almost entirely geared toward ML workloads.
dontwearitout 4y ago
I think the answer to this is that we don't anything that much better than GPUs to justify dedicated hardware - but I see it as inevitable. (Edit: I'm not counting TPUs, since they're not for sale). My money is on something like processing-in-memory enabling inference on gigantic models with rock bottom energy consumption, and rolling out to the masses as a PCIe device. Lots of research on this direction, see https://mythic.ai https://mythic.ai for an example.
- sroussey 4y ago
  See https://rain.ai https://rain.ai Neuromorphic hardware
  - seydor 4y ago
    I dont understand why pursuing "neuromorphic hardware". These are like asics running the most rudimentary (and wrong) simulation of human neurons, which are not known to solve any problem very well, in order to ... what exactly? Artificial neural networks with backprop is all we need for AI
    - nextaccountic 4y ago
      Probably to save energy, and maybe to get a performance boost.
    - BlueTemplar 4y ago
      Look at what happened with bitcoin : first mined on CPUs, then GPUs, but now only dedicated ASICs that can only run specific functions are competitive. Same thing is likely to happen here : artificial neural networks directly in hardware might have extra possibilities for optimization ?
    - dontwearitout 4y ago
      >Artificial neural networks with backprop is all we need for AI No. Biological neural networks run many orders of magnitude more energy efficiently, and learn substantially more data efficiently and continuously. We've got a lot of catching up to do with nature. In my opinion, backprop on today's compute architecture won't ever compete (I can elaborate if you like) - we need some fundamental changes in our approach.
potatoman22 4y ago
The Intel Neural Compute Stick is like this. It's a USB stick intended to power computer vision and other ML tasks. It's pretty tiny though and isn't nearly as powerful as a graphics card.
machiaweliczny 4y ago
There's very interesting and easy implementation for Tsetlin Machines in hardware that I think have possibility to replace transformers (I am not in field though so might be wrong).
dragonwriter 4y ago
> I wonder why we don't have external "neural processing" devices like we once had soundcards Well, we have dedicated coprocessors for them (Google TPUs, Nvidia Tensor Cores, Apple Neural Engine, etc.) There are dedicated cards, though some systems come with them on the mainboard and some videocards have them as well.

by seydor 4y ago

jhrmnn 4y ago

https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit

seydor 4y ago
but those are not for sale, and not transformer-specific. There must be some optimizations that can be done in hardware and transformers are several years old now
- ruuda 4y ago
  You likely already bought one. https://blog.google/products/pixel/introducing-google-tensor/ https://blog.google/products/pixel/introducing-google-tensor... https://apple.fandom.com/wiki/Neural_Engine https://apple.fandom.com/wiki/Neural_Engine
- jhrmnn 4y ago
  Computation-wise, transformers are really just a bunch of matrix multiplications, nothing more to it. (Which is partially why they’re so efficient and scalable.) Also, Nvidia’s GPU architectures are moving in the TPU direction (https://www.nvidia.com/en-us/data-center/tensor-cores/ https://www.nvidia.com/en-us/data-center/tensor-cores/).
  - shaklee3 4y ago
    Nvidia had tensor cores before the tpu existed
    - jhrmnn 4y ago
      Right, correct. I was already thinking ahead about the recent TensorFloat precision, which is further directly targeting matrix multiplication.
    - dragonwriter 4y ago
      > Nvidia had tensor cores before the tpu existed TPUs were released in 2016, Tensor Cores in 2017. But Edge TPUs (that is, the ones that exist outside of Google's Cloud infrastructure) weren’t until 2018. So, Nvidia had their cores available for people to buy first, but not before TPUs existed.
- dragonwriter 4y ago
  Google TPU are for sale [0] as are Nvidia cards with their Tensor Cores. As are Apple mainboards with their Neural Engine. [0] https://coral.ai/products/ https://coral.ai/products/

zenogantner 4y ago

> wonder why we don't have external "neural processing" devices like we once had soundcards. Some video cards/GPUs have become just that, becoming more and more geared towards non-graphics workloads ...

valine 4y ago

Nvidia A100 is exactly that. It has lower cuda performance than a RTX 4090, and is almost entirely geared toward ML workloads.

dontwearitout 4y ago

I think the answer to this is that we don't anything that much better than GPUs to justify dedicated hardware - but I see it as inevitable. (Edit: I'm not counting TPUs, since they're not for sale). My money is on something like processing-in-memory enabling inference on gigantic models with rock bottom energy consumption, and rolling out to the masses as a PCIe device. Lots of research on this direction, see https://mythic.ai https://mythic.ai for an example.

sroussey 4y ago
See https://rain.ai https://rain.ai Neuromorphic hardware
- seydor 4y ago
  I dont understand why pursuing "neuromorphic hardware". These are like asics running the most rudimentary (and wrong) simulation of human neurons, which are not known to solve any problem very well, in order to ... what exactly? Artificial neural networks with backprop is all we need for AI
  - nextaccountic 4y ago
    Probably to save energy, and maybe to get a performance boost.
  - BlueTemplar 4y ago
    Look at what happened with bitcoin : first mined on CPUs, then GPUs, but now only dedicated ASICs that can only run specific functions are competitive. Same thing is likely to happen here : artificial neural networks directly in hardware might have extra possibilities for optimization ?
  - dontwearitout 4y ago
    >Artificial neural networks with backprop is all we need for AI No. Biological neural networks run many orders of magnitude more energy efficiently, and learn substantially more data efficiently and continuously. We've got a lot of catching up to do with nature. In my opinion, backprop on today's compute architecture won't ever compete (I can elaborate if you like) - we need some fundamental changes in our approach.

potatoman22 4y ago

The Intel Neural Compute Stick is like this. It's a USB stick intended to power computer vision and other ML tasks. It's pretty tiny though and isn't nearly as powerful as a graphics card.

machiaweliczny 4y ago

There's very interesting and easy implementation for Tsetlin Machines in hardware that I think have possibility to replace transformers (I am not in field though so might be wrong).

dragonwriter 4y ago

> I wonder why we don't have external "neural processing" devices like we once had soundcards Well, we have dedicated coprocessors for them (Google TPUs, Nvidia Tensor Cores, Apple Neural Engine, etc.) There are dedicated cards, though some systems come with them on the mainboard and some videocards have them as well.