6 ms·

Event-based backpropagation for exact gradients in spiking neural networks

119 points by berndi 5y ago

periheli0n 5y ago
This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years. But IMHO it's time that field moved on from running after matrix accelerators and focused on the real advantages of event-based computing: asynchronous, low-latency, event-based signal processing.
- mpfundstein 5y ago
  interesting. where would one start reading about all this?
  - periheli0n 5y ago
    You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the research papers to find out that the field has failed to live up to the promise of lower-energy deep learning (which was a misguided promise from the outset, IMHO).
    - sroussey 5y ago
      Does that include rain.ai ?
      - periheli0n 5y ago
        I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
      - wombat23 5y ago
        Looks like a different approach. Intel's chip is based on digital circuits. They try an analog approach.
    - dtjohnnyb 5y ago
      Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
      - periheli0n 5y ago
        Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
        orbifold 5y ago
        I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does not mean that it can't be sparse. Since you are actually processing one spike at a time (potentially asynchronously), you don't need to implement any matrix multiplication. Current neuromorphic hardware achieves at least some degree of sparsity in their synaptic crossbars (BrainScales2, Spinnaker) or largely eliminates them like Loihi.
        periheli0n 5y ago
        Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency code costs time, in terms of computation, and also in terms of representation, before even one item is classified. Perceptron-based ANNs with continuous outputs don't require this step and will always have an edge over spiking networks in such scenarios. I think what the field is really lacking is an ML problem that can leverage spiking networks directly, that does not require costly conversion of data into a representation that is suitable for spiking networks.
  - clara732 5y ago
    Here is a paper from the same group which includes actual results of an algorithm running on the neuromorphic chip: https://arxiv.org/abs/1912.11443 https://arxiv.org/abs/1912.11443
  - marmaduke 5y ago
    The EU Brainscales project built a wafer that runs 10k times faster than real-time, https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_hardware_configuration.html https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_...
- FrereKhan 5y ago
  It's not quite correct to say this is only for achieving deep learning. Gradient-based parameter optimisation is still a useful tool, even for small shallow networks that would be ideal for event-based signal processing. Even for small-network tasks, training spiking networks has been non-trivial. This paper provides a way to get exact gradients, implying probably faster optimisation than using surrogate gradients or other approximation methods for SNNs.
  - periheli0n 5y ago
    You are totally right. The algorithm itself is a potential game-changer. I guess I was carried away by the pitch in the abstract that starts off with deep learning. Personally I think that way too many resources were wasted on trying to make better deep networks with spikes. In my opinion it is much more promising to apply spiking networks on problems that are inherently event-based. Having a functional backpropagation algorithm such as the one provided can help with that, obviously.
    - datameta 5y ago
      Based on reading just the abstract so far, it seems to me the event-based application of this algorithm makes absolute sense. Temporal importance can be effectively characterized in memristors. At the risk of making a comparison similar to that of Andrew Ng's a decade ago, I think this approach paired with something like a ReRAM crossbar is quite an effective rough analogue to the voltage potentials across a group of neurons in the brain. I applaud this team's efforts. A real breakthrough.
orbifold 5y ago
One of the authors here, happy to answer any questions you might have.
- haffi112 5y ago
  In table 3 you compare your result with other publications. Reference 44 has a larger accuracy than you with fewer hidden neurons. What is the difference between your method and theirs? Also, will you release your method as code?
  - orbifold 5y ago
    That number is what they reported in their publication, but it turns out they actually used both recurrent neurons (compared to feedforward as they state) and 512 instead of 100 neurons (see here https://zenkelab.org/publications/errata_zenke_vogels_2021/ https://zenkelab.org/publications/errata_zenke_vogels_2021/). We will adjust those numbers in the final publication. My aim is to release the method as part of Norse https://github.com/norse/norse https://github.com/norse/norse. There is some subtlety involved in implementing it for a given integration scheme, though. The event based simulator underlying the paper will also be released in due time.
  - throwaway245234 5y ago
    Other author here, as was pointed out this was a typo in the cited publication. We will definitely be releasing the source code in due time.
- nairboon 5y ago
  Could you comment on this general SNN critique? https://news.ycombinator.com/item?id=27366059 https://news.ycombinator.com/item?id=27366059
  - periheli0n 5y ago
    As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs. Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an important problem in spiking network research. Whether spiking networks will provide lower-energy deep learning is a totally different question.
drdeca 5y ago
I'm unclear on what \tau_{syn} and \tau_{mem} mean. I assume that syn stands for synapse, and mem stands for memory, but I'm not sure what the \tau is about? Time? I imagine that this would be clear to someone in the field, who would be the target audience for the paper, so this isn't really a criticism.
- jegp 5y ago
  \tau is a common symbol for time constants which, briefly put, determines how fast something decays over time. Syn for synaptic current and mem for membrane voltage. We've written some documentation around our neuron equations in Python that explains this: https://norse.github.io/norse/auto_api/norse.torch.functional.leaky_integrator.html https://norse.github.io/norse/auto_api/norse.torch.functiona... See also our tutorial on neuron parameter optimization to understand how it's useful for machine learning: https://github.com/norse/notebooks#level-intermediate https://github.com/norse/notebooks#level-intermediate There's also a great book on the topic by Gerstner available online: https://neuronaldynamics.epfl.ch/ https://neuronaldynamics.epfl.ch/ Disclaimer: I'm a co-author of the library Norse Regarding the target audience, it's actually not entirely clear to me. This lies in the intersection between computational neuroscience and deep learning, which isn't a huge set of people. So, I think you're question is highly relevant and we (as researchers) have a lot of work in front of us to explain why this is interesting and important.
  - drdeca 5y ago
    Thanks! I will take a look at those

119 points by berndi 5y ago

This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years. But IMHO it's time that field moved on from running after matrix accelerators and focused on the real advantages of event-based computing: asynchronous, low-latency, event-based signal processing.

mpfundstein 5y ago
interesting. where would one start reading about all this?
- periheli0n 5y ago
  You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the research papers to find out that the field has failed to live up to the promise of lower-energy deep learning (which was a misguided promise from the outset, IMHO).
  - sroussey 5y ago
    Does that include rain.ai ?
    - periheli0n 5y ago
      I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
    - wombat23 5y ago
      Looks like a different approach. Intel's chip is based on digital circuits. They try an analog approach.
  - dtjohnnyb 5y ago
    Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
    - periheli0n 5y ago
      Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
      - orbifold 5y ago
        I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does not mean that it can't be sparse. Since you are actually processing one spike at a time (potentially asynchronously), you don't need to implement any matrix multiplication. Current neuromorphic hardware achieves at least some degree of sparsity in their synaptic crossbars (BrainScales2, Spinnaker) or largely eliminates them like Loihi.
        periheli0n 5y ago
        Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency code costs time, in terms of computation, and also in terms of representation, before even one item is classified. Perceptron-based ANNs with continuous outputs don't require this step and will always have an edge over spiking networks in such scenarios. I think what the field is really lacking is an ML problem that can leverage spiking networks directly, that does not require costly conversion of data into a representation that is suitable for spiking networks.
- clara732 5y ago
  Here is a paper from the same group which includes actual results of an algorithm running on the neuromorphic chip: https://arxiv.org/abs/1912.11443 https://arxiv.org/abs/1912.11443
- marmaduke 5y ago
  The EU Brainscales project built a wafer that runs 10k times faster than real-time, https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_hardware_configuration.html https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_...
FrereKhan 5y ago
It's not quite correct to say this is only for achieving deep learning. Gradient-based parameter optimisation is still a useful tool, even for small shallow networks that would be ideal for event-based signal processing. Even for small-network tasks, training spiking networks has been non-trivial. This paper provides a way to get exact gradients, implying probably faster optimisation than using surrogate gradients or other approximation methods for SNNs.
- periheli0n 5y ago
  You are totally right. The algorithm itself is a potential game-changer. I guess I was carried away by the pitch in the abstract that starts off with deep learning. Personally I think that way too many resources were wasted on trying to make better deep networks with spikes. In my opinion it is much more promising to apply spiking networks on problems that are inherently event-based. Having a functional backpropagation algorithm such as the one provided can help with that, obviously.
  - datameta 5y ago
    Based on reading just the abstract so far, it seems to me the event-based application of this algorithm makes absolute sense. Temporal importance can be effectively characterized in memristors. At the risk of making a comparison similar to that of Andrew Ng's a decade ago, I think this approach paired with something like a ReRAM crossbar is quite an effective rough analogue to the voltage potentials across a group of neurons in the brain. I applaud this team's efforts. A real breakthrough.

orbifold 5y ago

One of the authors here, happy to answer any questions you might have.

haffi112 5y ago
In table 3 you compare your result with other publications. Reference 44 has a larger accuracy than you with fewer hidden neurons. What is the difference between your method and theirs? Also, will you release your method as code?
- orbifold 5y ago
  That number is what they reported in their publication, but it turns out they actually used both recurrent neurons (compared to feedforward as they state) and 512 instead of 100 neurons (see here https://zenkelab.org/publications/errata_zenke_vogels_2021/ https://zenkelab.org/publications/errata_zenke_vogels_2021/). We will adjust those numbers in the final publication. My aim is to release the method as part of Norse https://github.com/norse/norse https://github.com/norse/norse. There is some subtlety involved in implementing it for a given integration scheme, though. The event based simulator underlying the paper will also be released in due time.
- throwaway245234 5y ago
  Other author here, as was pointed out this was a typo in the cited publication. We will definitely be releasing the source code in due time.
nairboon 5y ago
Could you comment on this general SNN critique? https://news.ycombinator.com/item?id=27366059 https://news.ycombinator.com/item?id=27366059
- periheli0n 5y ago
  As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs. Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an important problem in spiking network research. Whether spiking networks will provide lower-energy deep learning is a totally different question.

drdeca 5y ago

I'm unclear on what \tau_{syn} and \tau_{mem} mean. I assume that syn stands for synapse, and mem stands for memory, but I'm not sure what the \tau is about? Time? I imagine that this would be clear to someone in the field, who would be the target audience for the paper, so this isn't really a criticism.

jegp 5y ago
\tau is a common symbol for time constants which, briefly put, determines how fast something decays over time. Syn for synaptic current and mem for membrane voltage. We've written some documentation around our neuron equations in Python that explains this: https://norse.github.io/norse/auto_api/norse.torch.functional.leaky_integrator.html https://norse.github.io/norse/auto_api/norse.torch.functiona... See also our tutorial on neuron parameter optimization to understand how it's useful for machine learning: https://github.com/norse/notebooks#level-intermediate https://github.com/norse/notebooks#level-intermediate There's also a great book on the topic by Gerstner available online: https://neuronaldynamics.epfl.ch/ https://neuronaldynamics.epfl.ch/ Disclaimer: I'm a co-author of the library Norse Regarding the target audience, it's actually not entirely clear to me. This lies in the intersection between computational neuroscience and deep learning, which isn't a huge set of people. So, I think you're question is highly relevant and we (as researchers) have a lot of work in front of us to explain why this is interesting and important.
- drdeca 5y ago
  Thanks! I will take a look at those