3 ms·

They are more power efficient when implemented as analogue circuits. Two main reasons: First, you only need a handful of transistors to implement a single spik

They are more power efficient when implemented as analogue circuits. Two main reasons: First, you only need a handful of transistors to implement a single spiking neuron, whereas digital circuits require ten to hundred thousands of transistors for corresponding numerical computations. Second, spiking neural networks are intrinsically asynchronous. All the computing elements (neurons) are independent and only communicate via rare one-bit events (spikes). Correspondingly you (if any) only need a clock for the event bus but not for the vast majority of the chip. See http://web.stanford.edu/group/brainsinsilicon/documents/IEEE2017.pdf http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... for a high-level overview. Edit: To clarify: the asynchronicity argument is valid for digital implementations of spiking neural networks as well, especially as you scale up to networks of a million (or more) neurons. See http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ for an implementation.

p1esk 9y ago
Why are you comparing analog spiking hardware to digital non-spiking hardware?
- shard 9y ago
  He is answering the question that his parent post asked, which is how can a spiking network be more efficient than a non-spiking one. He provided the details to a specific case of spiking networks which are more efficient, and explanations as to why it is more efficient.
  - p1esk 9y ago
    It should be obvious that analog hardware is more efficient than digital one. No explanation is needed there. I, on the other hand, would like to know how spiking helps when dealing with dense input signal matrices of common deep learning applications. It seems to me that analog non-spiking hardware is better suited for those tasks, especially when you're able to arrange processing elements in a large crossbar array, and colocate memory and processing (e.g. weights and multipliers) in the same element. Representing a signal value (of a certain precision) using a constant voltage level, or a current magnitude seems to be easier, more natural and more efficient than using either timings between spikes, or spike trains (spiking rate encoding the signal).
    - aurelian15 9y ago
      You're right that analogue non-spiking hardware in theory would be the way to go for deep neural networks. However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals. A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron. Edit: Reformulated some parts for clarity
      - p1esk 9y ago
        In an ideal world, yes a spike would be a simple binary signal, easy to generate/detect. Unfortunately, in the world we live in, it's quite a bit more complicated. Have you ever looked at Spice simulations where you needed to propagate a pulse (a spike with duration of one clock cycle)? It quickly gets distorted and attenuated. I don't see how noise in the circuit is anything less of a problem for a spike than it is for a constant signal. It actually seems to be more susceptible to noise, because now some spurious noise injection could be interpreted as a spike by the receiver! What makes you think short pulses are more noise resistant than long ones? And if you try to utilize interspike timings to encode information you expose yourself to a whole bunch of additional challenges (not sure if those timings are used in the current spiking network models though).
        aurelian15 9y ago
        In analogue hardware implementations spike events are transmitted on a standard digital bus using AER (address-event representation). The signal is not attenuated, since standard digital hardware (though asynchronous, depending on the implementation) is used to transmit the spike. Spurious noise injection is thus not a significant problem as well. Note that I'm only a layman when it comes to analogue neuromorphic hardware implementation details, I encourage you to have a look at [1] for more detailed energy computations. [1] http://web.stanford.edu/group/brainsinsilicon/documents/IEEE2017.pdf http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... Edit: Added a "significant" above. There indeed are minor problems with cross talk causing additional spikes, but mostly in the analogue neuron subthreshold regime, not in the AER bus. [2] [2] http://journal.frontiersin.org/article/10.3389/fncom.2017.00071/full http://journal.frontiersin.org/article/10.3389/fncom.2017.00...

p1esk 9y ago

Why are you comparing analog spiking hardware to digital non-spiking hardware?

shard 9y ago
He is answering the question that his parent post asked, which is how can a spiking network be more efficient than a non-spiking one. He provided the details to a specific case of spiking networks which are more efficient, and explanations as to why it is more efficient.
- p1esk 9y ago
  It should be obvious that analog hardware is more efficient than digital one. No explanation is needed there. I, on the other hand, would like to know how spiking helps when dealing with dense input signal matrices of common deep learning applications. It seems to me that analog non-spiking hardware is better suited for those tasks, especially when you're able to arrange processing elements in a large crossbar array, and colocate memory and processing (e.g. weights and multipliers) in the same element. Representing a signal value (of a certain precision) using a constant voltage level, or a current magnitude seems to be easier, more natural and more efficient than using either timings between spikes, or spike trains (spiking rate encoding the signal).
  - aurelian15 9y ago
    You're right that analogue non-spiking hardware in theory would be the way to go for deep neural networks. However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals. A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron. Edit: Reformulated some parts for clarity
    - p1esk 9y ago
      In an ideal world, yes a spike would be a simple binary signal, easy to generate/detect. Unfortunately, in the world we live in, it's quite a bit more complicated. Have you ever looked at Spice simulations where you needed to propagate a pulse (a spike with duration of one clock cycle)? It quickly gets distorted and attenuated. I don't see how noise in the circuit is anything less of a problem for a spike than it is for a constant signal. It actually seems to be more susceptible to noise, because now some spurious noise injection could be interpreted as a spike by the receiver! What makes you think short pulses are more noise resistant than long ones? And if you try to utilize interspike timings to encode information you expose yourself to a whole bunch of additional challenges (not sure if those timings are used in the current spiking network models though).
      - aurelian15 9y ago
        In analogue hardware implementations spike events are transmitted on a standard digital bus using AER (address-event representation). The signal is not attenuated, since standard digital hardware (though asynchronous, depending on the implementation) is used to transmit the spike. Spurious noise injection is thus not a significant problem as well. Note that I'm only a layman when it comes to analogue neuromorphic hardware implementation details, I encourage you to have a look at [1] for more detailed energy computations. [1] http://web.stanford.edu/group/brainsinsilicon/documents/IEEE2017.pdf http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... Edit: Added a "significant" above. There indeed are minor problems with cross talk causing additional spikes, but mostly in the analogue neuron subthreshold regime, not in the AER bus. [2] [2] http://journal.frontiersin.org/article/10.3389/fncom.2017.00071/full http://journal.frontiersin.org/article/10.3389/fncom.2017.00...