5 ms·

I'm from the lab working on this platform. A lot of people in this thread are claiming that there is no use for these spiking neural nets and I have to disagree

I'm from the lab working on this platform. A lot of people in this thread are claiming that there is no use for these spiking neural nets and I have to disagree. Brainstorm can run Spaun, the world's largest functioning and behaving brain model. [1] It can also run Adaptive Control circuits, which lead to unprecedented adaptability in arm control to undefined forces, such as changes in arm length, medium the arm is acting in and weights applied to the arm. [2] For an overview of how programming with spiking neural networks is different than normal programming, see this blog post I wrote almost a year ago. [3] Finally, framing spiking neural networks in opposition to Deep Learning networks is innacurate and harmful. We're currently investigating biologically plausible backprop and have already converted ConvNets to spiking nets for significant power savings [4]. Personally, I'm looking forward to an integration of biologically plausible networks with Deep Nets to completely theoretically unite all the approaches to brain modelling in general so we are no longer limited by metaphors [5]. [1] http://science.sciencemag.org/content/338/6111/1202 http://science.sciencemag.org/content/338/6111/1202 [2] http://rspb.royalsocietypublishing.org/content/283/1843/20162134 http://rspb.royalsocietypublishing.org/content/283/1843/2016... [3] https://medium.com/@seanaubin/a-way-around-the-coming-performance-walls-neuromorphic-hardware-with-spiking-neurons-facd4291b201 https://medium.com/@seanaubin/a-way-around-the-coming-perfor... [4] https://arxiv.org/abs/1611.05141 https://arxiv.org/abs/1611.05141 [5] https://medium.com/@seanaubin/deep-learning-is-almost-the-brain-3aaecd924f3d https://medium.com/@seanaubin/deep-learning-is-almost-the-br...

p1esk 10y ago
Hi Sean, perhaps you can help us understand the claim that analog hardware SNNs are more energy efficient than analog hardware non-spiking networks (e.g. based on floating gate transistors [1])? Also, in the paper [2], Table 4 is a bit confusing: what is the "traditional hardware" they're referring to? Finally, it's not clear from that paper, or from this "Brainstorm" article, that you actually have a working chip, capable of running Alexnet. Is that so? What is the current status of the actual hardware? [1] https://arxiv.org/abs/1610.02091 https://arxiv.org/abs/1610.02091 [2] https://arxiv.org/abs/1611.05141 https://arxiv.org/abs/1611.05141
- Seanny123 10y ago
  I'm not sure it's fair to compare the Brainstorm chip to the paper you linked to, since they take different inputs (binary vs. continuous) and they implement different systems (purely feed-forward vs. recurrent dynamic systems). But basically, the reason I would imagine why rates require more energy than spikes is just basic math. Given that power is the area under the curve, you can picture that intermittent spikes would consume less power than sending a continuous rate signal. There are other advantages to spikes vs. rates, such as robustness to noise/failure. In table 4, I think they're comparing to GPU hardware, since that's what he was running everything on. I think the current status of the hardware is prototypes are being produced. The most recent paper [1] used a circuit simulated in SPICE, so I'm guessing they're pretty close to production? I'm not sure, because: a) I haven't heard anything from my lab-mates in a while. b) Even if I did hear something from them, I'm not sure I'm allowed to talk about it. Hardware development is way more secretive than software development. [1] http://compneuro.uwaterloo.ca/files/publications/voelker.2017a.pdf http://compneuro.uwaterloo.ca/files/publications/voelker.201...
  - p1esk 10y ago
    Sure, let's look at the "area under the curve". What would be this area to perform an equivalent of a weighted sum followed by a non-linearity, for the best spiking network? Note that we must ensure the same precision, as when using the analog "continuous rate" signals. If SNN produces less accurate results, then it's really apples to oranges comparison. Don't forget to take into account both static and dynamic power. Also, there's a question of speed: can you build an SNN chip which runs faster given the same power budget, and the same target accuracy? If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! 3 times better efficiency than a 32 bit FLOPs digital GPU (which is probably 2-3 generations old by now)? That's not going to impress anyone. AFAIK, the best digital DL chips are already at least 10 times more efficient than latest GPUs, and analog DL chips claim to have at least another order of magnitude improvement. In my opinion, that's still not enough: if you want to build an analog chip (spiking or not), it better be at least 1000 times more power efficient than the best we can expect from Nvidia in the next couple of years. Otherwise it's just not worth the effort (and inflexibility). So, the bottom line, until there's a working chip, capable of running Alexnet, we have no guarantees about it's overall energy efficiency, or speed, or accuracy, or noise robustness. Simulating a tiny portion of it in SPICE does not really provide much insight. When the chip is built, and working, then we can compare it to the best analog "continuous rate" chip running the same model, and only then we will be able to see which one is more efficient. Until that time, any claims that spikes are more efficient are unsubstantiated. On the other hand, if you can devise an algorithm which is uniquely suited for spiking networks (biologically plausible backprop, or whatever), then sure, it's quite possible that you will be able to do things more efficiently. So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? That's not a rhetorical question, perhaps there are good reasons, and I'd like to know them.

p1esk 10y ago

Hi Sean, perhaps you can help us understand the claim that analog hardware SNNs are more energy efficient than analog hardware non-spiking networks (e.g. based on floating gate transistors [1])? Also, in the paper [2], Table 4 is a bit confusing: what is the "traditional hardware" they're referring to? Finally, it's not clear from that paper, or from this "Brainstorm" article, that you actually have a working chip, capable of running Alexnet. Is that so? What is the current status of the actual hardware? [1] https://arxiv.org/abs/1610.02091 https://arxiv.org/abs/1610.02091 [2] https://arxiv.org/abs/1611.05141 https://arxiv.org/abs/1611.05141

Seanny123 10y ago
I'm not sure it's fair to compare the Brainstorm chip to the paper you linked to, since they take different inputs (binary vs. continuous) and they implement different systems (purely feed-forward vs. recurrent dynamic systems). But basically, the reason I would imagine why rates require more energy than spikes is just basic math. Given that power is the area under the curve, you can picture that intermittent spikes would consume less power than sending a continuous rate signal. There are other advantages to spikes vs. rates, such as robustness to noise/failure. In table 4, I think they're comparing to GPU hardware, since that's what he was running everything on. I think the current status of the hardware is prototypes are being produced. The most recent paper [1] used a circuit simulated in SPICE, so I'm guessing they're pretty close to production? I'm not sure, because: a) I haven't heard anything from my lab-mates in a while. b) Even if I did hear something from them, I'm not sure I'm allowed to talk about it. Hardware development is way more secretive than software development. [1] http://compneuro.uwaterloo.ca/files/publications/voelker.2017a.pdf http://compneuro.uwaterloo.ca/files/publications/voelker.201...
- p1esk 10y ago
  Sure, let's look at the "area under the curve". What would be this area to perform an equivalent of a weighted sum followed by a non-linearity, for the best spiking network? Note that we must ensure the same precision, as when using the analog "continuous rate" signals. If SNN produces less accurate results, then it's really apples to oranges comparison. Don't forget to take into account both static and dynamic power. Also, there's a question of speed: can you build an SNN chip which runs faster given the same power budget, and the same target accuracy? If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! 3 times better efficiency than a 32 bit FLOPs digital GPU (which is probably 2-3 generations old by now)? That's not going to impress anyone. AFAIK, the best digital DL chips are already at least 10 times more efficient than latest GPUs, and analog DL chips claim to have at least another order of magnitude improvement. In my opinion, that's still not enough: if you want to build an analog chip (spiking or not), it better be at least 1000 times more power efficient than the best we can expect from Nvidia in the next couple of years. Otherwise it's just not worth the effort (and inflexibility). So, the bottom line, until there's a working chip, capable of running Alexnet, we have no guarantees about it's overall energy efficiency, or speed, or accuracy, or noise robustness. Simulating a tiny portion of it in SPICE does not really provide much insight. When the chip is built, and working, then we can compare it to the best analog "continuous rate" chip running the same model, and only then we will be able to see which one is more efficient. Until that time, any claims that spikes are more efficient are unsubstantiated. On the other hand, if you can devise an algorithm which is uniquely suited for spiking networks (biologically plausible backprop, or whatever), then sure, it's quite possible that you will be able to do things more efficiently. So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? That's not a rhetorical question, perhaps there are good reasons, and I'd like to know them.