3 ms·
I think the answer to this is that we don't anything that much better than GPUs to justify dedicated hardware - but I see it as inevitable. (Edit: I'm not count
by dontwearitout 4y ago
I think the answer to this is that we don't anything that much better than GPUs to justify dedicated hardware - but I see it as inevitable. (Edit: I'm not counting TPUs, since they're not for sale).
My money is on something like processing-in-memory enabling inference on gigantic models with rock bottom energy consumption, and rolling out to the masses as a PCIe device. Lots of research on this direction, see https://mythic.ai https://mythic.ai for an example.
- sroussey 4y agoSee https://rain.ai https://rain.ai Neuromorphic hardware
- seydor 4y agoI dont understand why pursuing "neuromorphic hardware". These are like asics running the most rudimentary (and wrong) simulation of human neurons, which are not known to solve any problem very well, in order to ... what exactly? Artificial neural networks with backprop is all we need for AI
- nextaccountic 4y agoProbably to save energy, and maybe to get a performance boost.
- BlueTemplar 4y agoLook at what happened with bitcoin : first mined on CPUs, then GPUs, but now only dedicated ASICs that can only run specific functions are competitive. Same thing is likely to happen here : artificial neural networks directly in hardware might have extra possibilities for optimization ?
- dontwearitout 4y ago>Artificial neural networks with backprop is all we need for AI No. Biological neural networks run many orders of magnitude more energy efficiently, and learn substantially more data efficiently and continuously. We've got a lot of catching up to do with nature. In my opinion, backprop on today's compute architecture won't ever compete (I can elaborate if you like) - we need some fundamental changes in our approach.
- reasonabl_human 4y agoOkay I’ll bite - please elaborate!
- dontwearitout 4y agoMy opinion is heavily shaped by this interview (5-9 minutes is a good summary): https://www.youtube.com/watch?v=0PAiQ1jTN5k https://www.youtube.com/watch?v=0PAiQ1jTN5k. We're currently running up against the Von Neumann bottleneck in ML. It's a great architecture for general purpose compute, but not suited for efficient operation of neural networks: a ton of energy is spent shuttling weight and activation values between GPU RAM and ALUs. Biological networks appear to be able to process a large amount of information per unit energy by exploiting the static physical structure of the network. A small amount of energy in sparse neural activations can transform a large volume of information stored in the structure and synapse strengths. Here's some very rough napkin math: assume the brain has 100 billion neurons, each with 1000 synapses, and is sparsely activated at ~5% activation, and processes at 30Hz (gamma wave frequency, roughly). This means every "tick", 5 billion neurons must interact with 5 trillion other neurons. If you tried to implement this in a Von Neumann architecture, even if you only calculate for the sparsely activated neurons, and even if you quantize your weights values to 1 byte, this means processing ~150 TB/s of weight data. That's an insane amount of memory bandwidth. An A100 is 2 TB/s at 300W, but our brain only uses 20W. My two cents is that the future will involve a hardware architecture that let's us avoid moving these weights around during inference. Whether this will mean a "mortal computer" as Hinton has recently discussed (https://www.youtube.com/watch?v=sghvwkXV3VU https://www.youtube.com/watch?v=sghvwkXV3VU) or whether the weights will be loaded and fixed at init time (as with Mythic's or other neuromorphic approaches), time will tell.
- seydor 4y agoneuromorphic chips are not biological neural networks
- dontwearitout 4y agoVery exciting stuff, thanks for sharing this. They don't have a ton of open positions, but I'll be keeping an eye on the careers page.