4 ms·

interesting. where would one start reading about all this?

periheli0n 5y ago
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the research papers to find out that the field has failed to live up to the promise of lower-energy deep learning (which was a misguided promise from the outset, IMHO).
- sroussey 5y ago
  Does that include rain.ai ?
  - periheli0n 5y ago
    I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
  - wombat23 5y ago
    Looks like a different approach. Intel's chip is based on digital circuits. They try an analog approach.
- dtjohnnyb 5y ago
  Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
  - periheli0n 5y ago
    Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
    - orbifold 5y ago
      I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does not mean that it can't be sparse. Since you are actually processing one spike at a time (potentially asynchronously), you don't need to implement any matrix multiplication. Current neuromorphic hardware achieves at least some degree of sparsity in their synaptic crossbars (BrainScales2, Spinnaker) or largely eliminates them like Loihi.
      - periheli0n 5y ago
        Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency code costs time, in terms of computation, and also in terms of representation, before even one item is classified. Perceptron-based ANNs with continuous outputs don't require this step and will always have an edge over spiking networks in such scenarios. I think what the field is really lacking is an ML problem that can leverage spiking networks directly, that does not require costly conversion of data into a representation that is suitable for spiking networks.
clara732 5y ago
Here is a paper from the same group which includes actual results of an algorithm running on the neuromorphic chip: https://arxiv.org/abs/1912.11443 https://arxiv.org/abs/1912.11443
marmaduke 5y ago
The EU Brainscales project built a wafer that runs 10k times faster than real-time, https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_hardware_configuration.html https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_...

You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphic-computing.html https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the research papers to find out that the field has failed to live up to the promise of lower-energy deep learning (which was a misguided promise from the outset, IMHO).

sroussey 5y ago
Does that include rain.ai ?
- periheli0n 5y ago
  I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
- wombat23 5y ago
  Looks like a different approach. Intel's chip is based on digital circuits. They try an analog approach.
dtjohnnyb 5y ago
Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
- periheli0n 5y ago
  Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
  - orbifold 5y ago
    I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does not mean that it can't be sparse. Since you are actually processing one spike at a time (potentially asynchronously), you don't need to implement any matrix multiplication. Current neuromorphic hardware achieves at least some degree of sparsity in their synaptic crossbars (BrainScales2, Spinnaker) or largely eliminates them like Loihi.
    - periheli0n 5y ago
      Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency code costs time, in terms of computation, and also in terms of representation, before even one item is classified. Perceptron-based ANNs with continuous outputs don't require this step and will always have an edge over spiking networks in such scenarios. I think what the field is really lacking is an ML problem that can leverage spiking networks directly, that does not require costly conversion of data into a representation that is suitable for spiking networks.

clara732 5y ago

Here is a paper from the same group which includes actual results of an algorithm running on the neuromorphic chip: https://arxiv.org/abs/1912.11443 https://arxiv.org/abs/1912.11443

marmaduke 5y ago

The EU Brainscales project built a wafer that runs 10k times faster than real-time, https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_hardware_configuration.html https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_...