4 ms·

> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?

by dartos 3y ago

p1esk 3y ago
This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.
- imtringued 3y ago
  Spiking neural networks are not software, they are usually built directly into silicon chips because they are using pulse timing to encode information instead of multiple bits. The problem is that training them is difficult because they operate over time, not that they don't work. As of now, scaling training infrastructure is more important than theoretical power efficiency.
- colinator 3y ago
  Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part. I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?
  - ilaksh 3y ago
    But this group claims to have demonstrated a way to use SNNs to run LLMs effectively and with vastly less energy usage.
    - p1esk 3y ago
      If OpenAI or DeepMind makes such claim I'd pay attention. Otherwise it's always some (usually hw) guys trying to get a grant, or even just publish a paper. p.s. People interested in biologically inspired data processing algorithms should look at Numenta's papers (earlier ones, because recently they switched to regular deep learning), and especially learn their justification for not using spikes.
      - esafak 3y ago
        They switched? Why? There goes their whole raison d'etre!
zachbee 3y ago
Neuromorphic computing basically uses individual "neurons", represented with either analog or digital circuits, which communicate using asynchronous pulses called "spikes". Unlike the human brain, neuromorphic chips are 2D, but we can replicate a good amount of neural dynamics in silicon. It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures. https://web.stanford.edu/group/brainsinsilicon/documents/ANeckar2019.pdf https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/BenjaminEtAlNeurogrid2014.pdf https://web.stanford.edu/group/brainsinsilicon/documents/Ben...

> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?

by dartos 3y ago

> New structure mimics the layout of neurons and synapses What does that mean, practically? How can you mimic that layout in silicon?

p1esk 3y ago

This means they use Spiking Neural Networks. It’s a software algorithm that most likely doesn’t work as well as regular NNs.

imtringued 3y ago
Spiking neural networks are not software, they are usually built directly into silicon chips because they are using pulse timing to encode information instead of multiple bits. The problem is that training them is difficult because they operate over time, not that they don't work. As of now, scaling training infrastructure is more important than theoretical power efficiency.
colinator 3y ago
Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part. I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?
- ilaksh 3y ago
  But this group claims to have demonstrated a way to use SNNs to run LLMs effectively and with vastly less energy usage.
  - p1esk 3y ago
    If OpenAI or DeepMind makes such claim I'd pay attention. Otherwise it's always some (usually hw) guys trying to get a grant, or even just publish a paper. p.s. People interested in biologically inspired data processing algorithms should look at Numenta's papers (earlier ones, because recently they switched to regular deep learning), and especially learn their justification for not using spikes.
    - esafak 3y ago
      They switched? Why? There goes their whole raison d'etre!

zachbee 3y ago

Neuromorphic computing basically uses individual "neurons", represented with either analog or digital circuits, which communicate using asynchronous pulses called "spikes". Unlike the human brain, neuromorphic chips are 2D, but we can replicate a good amount of neural dynamics in silicon. It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures. https://web.stanford.edu/group/brainsinsilicon/documents/ANeckar2019.pdf https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/BenjaminEtAlNeurogrid2014.pdf https://web.stanford.edu/group/brainsinsilicon/documents/Ben...