5 ms·

This hardware is designed for running spiking neural network models? What is the current state of the art in this field? I was under the impression that trainin

This hardware is designed for running spiking neural network models? What is the current state of the art in this field? I was under the impression that training a spiking neural network was somewhat of an unsolved problem, because backprop doesn't easily apply. Anyone have information on this?

aurelian15 9y ago
There are several frameworks for the construction of spiking neural networks given an abstract mathematical description of the desired function. The most complete (note that I'm currently a student in the lab that developed this method) is the Neural Engineering Framework [1]. There is a Python reference implementation called Nengo [2] which can target several backends, including neuromorphic hardware. Training deep spiking neural networks is possible as well [3]. [1] http://compneuro.uwaterloo.ca/research/nef.html http://compneuro.uwaterloo.ca/research/nef.html [2] https://www.nengo.ai/ https://www.nengo.ai/ [3] https://arxiv.org/abs/1611.05141 https://arxiv.org/abs/1611.05141
- p1esk 9y ago
  There's this common claim that spiking networks are somehow more power efficient. Yet I have never seen any evidence to support it, or even any decent explanation why would it ever be the case.
  - aurelian15 9y ago
    They are more power efficient when implemented as analogue circuits. Two main reasons: First, you only need a handful of transistors to implement a single spiking neuron, whereas digital circuits require ten to hundred thousands of transistors for corresponding numerical computations. Second, spiking neural networks are intrinsically asynchronous. All the computing elements (neurons) are independent and only communicate via rare one-bit events (spikes). Correspondingly you (if any) only need a clock for the event bus but not for the vast majority of the chip. See http://web.stanford.edu/group/brainsinsilicon/documents/IEEE2017.pdf http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... for a high-level overview. Edit: To clarify: the asynchronicity argument is valid for digital implementations of spiking neural networks as well, especially as you scale up to networks of a million (or more) neurons. See http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ for an implementation.
    - p1esk 9y ago
      Why are you comparing analog spiking hardware to digital non-spiking hardware?
      - shard 9y ago
        He is answering the question that his parent post asked, which is how can a spiking network be more efficient than a non-spiking one. He provided the details to a specific case of spiking networks which are more efficient, and explanations as to why it is more efficient.
        p1esk 9y ago
        It should be obvious that analog hardware is more efficient than digital one. No explanation is needed there. I, on the other hand, would like to know how spiking helps when dealing with dense input signal matrices of common deep learning applications. It seems to me that analog non-spiking hardware is better suited for those tasks, especially when you're able to arrange processing elements in a large crossbar array, and colocate memory and processing (e.g. weights and multipliers) in the same element. Representing a signal value (of a certain precision) using a constant voltage level, or a current magnitude seems to be easier, more natural and more efficient than using either timings between spikes, or spike trains (spiking rate encoding the signal).
        aurelian15 9y ago
        You're right that analogue non-spiking hardware in theory would be the way to go for deep neural networks. However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals. A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron. Edit: Reformulated some parts for clarity
- Seanny123 9y ago
  I'm from the lab that you're citing with those links and I'm kind of amazed you're so up to date with our research. Why do you know so much about Nengo? Are you actually one of my lab-mates, but I just can't identify you from your username?
  - moh_maya 9y ago
    " (note that I'm currently a student in the lab that developed this method)". Lab mate?
    - Seanny123 9y ago
      Huh, apparently. This will be an interesting conversation at the next lab meeting.
IncRnd 9y ago
They likely do this in order to avoid competition and comparisons.
- aurelian15 9y ago
  Nah. There are legitimate reasons for doing this. Especially when spiking neurons are implemented using analogue circuits (which I could find no indication of in the press release however, but see [1,2,3,4] for examples) spiking neurons offer significant advantages, such as extremely low power consumption, intrinsic dynamics, and asynchronicity (which again, reduces power consumption by not requiring clocks). [1] https://brainscales.kip.uni-heidelberg.de/ https://brainscales.kip.uni-heidelberg.de/ [2] http://www.kip.uni-heidelberg.de/vision/spikey/ http://www.kip.uni-heidelberg.de/vision/spikey/ [3] https://web.stanford.edu/group/brainsinsilicon/neurogrid.html https://web.stanford.edu/group/brainsinsilicon/neurogrid.htm... [4] http://ncs.ethz.ch/ http://ncs.ethz.ch/
  - shard 9y ago
    I was looking for clarification as to whether the neurons are implemented as analog circuits as well, especially with the reference to Dr. Carver Mead in the beginning of the article, who has a long history of working on analog neuron circuits, but I did not see any explicit references to whether the neurons were digital or analog.
  - IncRnd 9y ago
    Thank you!
- baq 9y ago
  intel is not a charity. if they do something, they either have a customer lined up or they think there's money on the table in the market, just as it should be. bragging rights are low cash flow.
  - IncRnd 9y ago
    I didn't mention bragging rights. Your comment supports what I wrote.
venachescu 9y ago
I'm pretty surprised by all the focus on spiking neural networks too; they are really still much more of an academic topic and I have never heard of any real practical applications (yet). There are a number of methods of training these kinds of networks like Spike Timing Dependent Plasticity (STDP) which is essentially a reinforcement learning algorithm that increases the weights between neurons that spike often together and modulates the increases by a reward signal. However, all these methods are really more focused on replicating and modeling the biological phenomena and not being performant. In theory, spiking networks should be much more efficient, real neurons are able to take advantage of many non-linear effects to create incredibly complex analog to digital (spikes) computation in every cell using incredibly little energy - our brains only use about 20 watts of power. But, the current approach to CPUs and even GPUs means simulating all this stuff is ridiculously inefficient and ultimately looks nothing like the real thing.

aurelian15 9y ago

There are several frameworks for the construction of spiking neural networks given an abstract mathematical description of the desired function. The most complete (note that I'm currently a student in the lab that developed this method) is the Neural Engineering Framework [1]. There is a Python reference implementation called Nengo [2] which can target several backends, including neuromorphic hardware. Training deep spiking neural networks is possible as well [3]. [1] http://compneuro.uwaterloo.ca/research/nef.html http://compneuro.uwaterloo.ca/research/nef.html [2] https://www.nengo.ai/ https://www.nengo.ai/ [3] https://arxiv.org/abs/1611.05141 https://arxiv.org/abs/1611.05141

p1esk 9y ago
There's this common claim that spiking networks are somehow more power efficient. Yet I have never seen any evidence to support it, or even any decent explanation why would it ever be the case.
- aurelian15 9y ago
  They are more power efficient when implemented as analogue circuits. Two main reasons: First, you only need a handful of transistors to implement a single spiking neuron, whereas digital circuits require ten to hundred thousands of transistors for corresponding numerical computations. Second, spiking neural networks are intrinsically asynchronous. All the computing elements (neurons) are independent and only communicate via rare one-bit events (spikes). Correspondingly you (if any) only need a clock for the event bus but not for the vast majority of the chip. See http://web.stanford.edu/group/brainsinsilicon/documents/IEEE2017.pdf http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... for a high-level overview. Edit: To clarify: the asynchronicity argument is valid for digital implementations of spiking neural networks as well, especially as you scale up to networks of a million (or more) neurons. See http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ for an implementation.
  - p1esk 9y ago
    Why are you comparing analog spiking hardware to digital non-spiking hardware?
    - shard 9y ago
      He is answering the question that his parent post asked, which is how can a spiking network be more efficient than a non-spiking one. He provided the details to a specific case of spiking networks which are more efficient, and explanations as to why it is more efficient.
      - p1esk 9y ago
        It should be obvious that analog hardware is more efficient than digital one. No explanation is needed there. I, on the other hand, would like to know how spiking helps when dealing with dense input signal matrices of common deep learning applications. It seems to me that analog non-spiking hardware is better suited for those tasks, especially when you're able to arrange processing elements in a large crossbar array, and colocate memory and processing (e.g. weights and multipliers) in the same element. Representing a signal value (of a certain precision) using a constant voltage level, or a current magnitude seems to be easier, more natural and more efficient than using either timings between spikes, or spike trains (spiking rate encoding the signal).
        aurelian15 9y ago
        You're right that analogue non-spiking hardware in theory would be the way to go for deep neural networks. However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals. A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron. Edit: Reformulated some parts for clarity
Seanny123 9y ago
I'm from the lab that you're citing with those links and I'm kind of amazed you're so up to date with our research. Why do you know so much about Nengo? Are you actually one of my lab-mates, but I just can't identify you from your username?
- moh_maya 9y ago
  " (note that I'm currently a student in the lab that developed this method)". Lab mate?
  - Seanny123 9y ago
    Huh, apparently. This will be an interesting conversation at the next lab meeting.

IncRnd 9y ago

They likely do this in order to avoid competition and comparisons.

aurelian15 9y ago
Nah. There are legitimate reasons for doing this. Especially when spiking neurons are implemented using analogue circuits (which I could find no indication of in the press release however, but see [1,2,3,4] for examples) spiking neurons offer significant advantages, such as extremely low power consumption, intrinsic dynamics, and asynchronicity (which again, reduces power consumption by not requiring clocks). [1] https://brainscales.kip.uni-heidelberg.de/ https://brainscales.kip.uni-heidelberg.de/ [2] http://www.kip.uni-heidelberg.de/vision/spikey/ http://www.kip.uni-heidelberg.de/vision/spikey/ [3] https://web.stanford.edu/group/brainsinsilicon/neurogrid.html https://web.stanford.edu/group/brainsinsilicon/neurogrid.htm... [4] http://ncs.ethz.ch/ http://ncs.ethz.ch/
- shard 9y ago
  I was looking for clarification as to whether the neurons are implemented as analog circuits as well, especially with the reference to Dr. Carver Mead in the beginning of the article, who has a long history of working on analog neuron circuits, but I did not see any explicit references to whether the neurons were digital or analog.
- IncRnd 9y ago
  Thank you!
baq 9y ago
intel is not a charity. if they do something, they either have a customer lined up or they think there's money on the table in the market, just as it should be. bragging rights are low cash flow.
- IncRnd 9y ago
  I didn't mention bragging rights. Your comment supports what I wrote.

venachescu 9y ago

I'm pretty surprised by all the focus on spiking neural networks too; they are really still much more of an academic topic and I have never heard of any real practical applications (yet). There are a number of methods of training these kinds of networks like Spike Timing Dependent Plasticity (STDP) which is essentially a reinforcement learning algorithm that increases the weights between neurons that spike often together and modulates the increases by a reward signal. However, all these methods are really more focused on replicating and modeling the biological phenomena and not being performant. In theory, spiking networks should be much more efficient, real neurons are able to take advantage of many non-linear effects to create incredibly complex analog to digital (spikes) computation in every cell using incredibly little energy - our brains only use about 20 watts of power. But, the current approach to CPUs and even GPUs means simulating all this stuff is ridiculously inefficient and ultimately looks nothing like the real thing.