5 ms·

I'm not sure it's fair to compare the Brainstorm chip to the paper you linked to, since they take different inputs (binary vs. continuous) and they implement di

I'm not sure it's fair to compare the Brainstorm chip to the paper you linked to, since they take different inputs (binary vs. continuous) and they implement different systems (purely feed-forward vs. recurrent dynamic systems). But basically, the reason I would imagine why rates require more energy than spikes is just basic math. Given that power is the area under the curve, you can picture that intermittent spikes would consume less power than sending a continuous rate signal. There are other advantages to spikes vs. rates, such as robustness to noise/failure. In table 4, I think they're comparing to GPU hardware, since that's what he was running everything on. I think the current status of the hardware is prototypes are being produced. The most recent paper [1] used a circuit simulated in SPICE, so I'm guessing they're pretty close to production? I'm not sure, because: a) I haven't heard anything from my lab-mates in a while. b) Even if I did hear something from them, I'm not sure I'm allowed to talk about it. Hardware development is way more secretive than software development. [1] http://compneuro.uwaterloo.ca/files/publications/voelker.2017a.pdf http://compneuro.uwaterloo.ca/files/publications/voelker.201...

p1esk 10y ago
Sure, let's look at the "area under the curve". What would be this area to perform an equivalent of a weighted sum followed by a non-linearity, for the best spiking network? Note that we must ensure the same precision, as when using the analog "continuous rate" signals. If SNN produces less accurate results, then it's really apples to oranges comparison. Don't forget to take into account both static and dynamic power. Also, there's a question of speed: can you build an SNN chip which runs faster given the same power budget, and the same target accuracy? If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! 3 times better efficiency than a 32 bit FLOPs digital GPU (which is probably 2-3 generations old by now)? That's not going to impress anyone. AFAIK, the best digital DL chips are already at least 10 times more efficient than latest GPUs, and analog DL chips claim to have at least another order of magnitude improvement. In my opinion, that's still not enough: if you want to build an analog chip (spiking or not), it better be at least 1000 times more power efficient than the best we can expect from Nvidia in the next couple of years. Otherwise it's just not worth the effort (and inflexibility). So, the bottom line, until there's a working chip, capable of running Alexnet, we have no guarantees about it's overall energy efficiency, or speed, or accuracy, or noise robustness. Simulating a tiny portion of it in SPICE does not really provide much insight. When the chip is built, and working, then we can compare it to the best analog "continuous rate" chip running the same model, and only then we will be able to see which one is more efficient. Until that time, any claims that spikes are more efficient are unsubstantiated. On the other hand, if you can devise an algorithm which is uniquely suited for spiking networks (biologically plausible backprop, or whatever), then sure, it's quite possible that you will be able to do things more efficiently. So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? That's not a rhetorical question, perhaps there are good reasons, and I'd like to know them.
- Seanny123 10y ago
  That's totally fair to want to wait for a comparison until there's actual hardware produced. Especially for comparisons exclusively involving DL models. Make initial "area under the curve" argument was rhetorical and not sufficiently empirically founded. > If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! Yeah, the paper talks about how this is preliminary and bigger savings are expected for video. However, I must concede as before that without the analog hardware built there isn't much point discussing this. > So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? Eric Hunsberger [1] has been doing most of the work in this domain, so I'm going to be awkwardly paraphrasing my conversations with him. Eric wanted to make Spaun's [2] vision system better. To do that, he knew he was going to need ConvNets or at least build something off of them. So he started to see if he could bring ConvNets into the domain of SNNs to understand them better. Once he did that, he started looking into if he could learn the SNN ConvNets using biologically plausible back-prop [3] which is where he's at now. That's really only a branch of our research. To understand the brain, we have a lot of really different methods that are more based in Symbolicism, Bayesianism and Dynamicism. We do start in software [2], but software is slow, even on a GPU. When we get faster hardware, we're able to explore the algos more quickly. Also, we got funding to build/explore analog hardware from the Office of Naval Research, so that's where this project is investigating. To summarize, the DNN-to-SNN-adaptation algos aren't the only thing targeting this hardware, but a small slice of a family of algos that are defining the requirements of the hardware. (I hope this post confirms that I understood and accept your argument and isn't me having a case of "must have the last word"-ism) [1] http://compneuro.uwaterloo.ca/people/eric-hunsberger.html http://compneuro.uwaterloo.ca/people/eric-hunsberger.html [2] https://nengo.github.io/ https://nengo.github.io/ [3] http://cogsci.stackexchange.com/q/16269/4397 http://cogsci.stackexchange.com/q/16269/4397
  - p1esk 10y ago
    I appreciate your response, especially the backstory of running a convnet on SNN hardware. A couple of remarks: 1. Your own response to your stackexchange question: "The random synaptic feedback weights only accomplish back-propagation through one layer of neurons, thus severely limiting the depth of a network." Didn't they show in the paper how it could work through multiple layers? 2. software is slow, even on a GPU. When we get faster hardware, we're able to explore the algos more quickly I'm not sure what you're referring to by "faster hardware", but somehow I doubt you will beat a workstation with 8 GPUs in terms of speed, if your goal is to explore algos more quickly. More importantly, what if the next algo you want to explore does not map well to the hardware you built? For example, what if we realize that relative timings between spikes are important, and most of the computation is based on that, but your hardware was not designed to exploit this "race logic" principles? Suddenly your custom system became much less useful, while your GPUs will simulate that just fine. There's a possibility that instead of exploring the best or most plausible algorithms, you will limit yourself to algorithms which map well to your hardware. 3. What do you think about HTM theory by Numenta? They strive for biologically realistic computation, but they don't think spikes are important, and abstract them away in their code. p.s. the reason I'm involved in this discussion is I'm trying to decide whether to accept an internship offer to work on Bayesian inference algorithms for SNN chip.

p1esk 10y ago

Sure, let's look at the "area under the curve". What would be this area to perform an equivalent of a weighted sum followed by a non-linearity, for the best spiking network? Note that we must ensure the same precision, as when using the analog "continuous rate" signals. If SNN produces less accurate results, then it's really apples to oranges comparison. Don't forget to take into account both static and dynamic power. Also, there's a question of speed: can you build an SNN chip which runs faster given the same power budget, and the same target accuracy? If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! 3 times better efficiency than a 32 bit FLOPs digital GPU (which is probably 2-3 generations old by now)? That's not going to impress anyone. AFAIK, the best digital DL chips are already at least 10 times more efficient than latest GPUs, and analog DL chips claim to have at least another order of magnitude improvement. In my opinion, that's still not enough: if you want to build an analog chip (spiking or not), it better be at least 1000 times more power efficient than the best we can expect from Nvidia in the next couple of years. Otherwise it's just not worth the effort (and inflexibility). So, the bottom line, until there's a working chip, capable of running Alexnet, we have no guarantees about it's overall energy efficiency, or speed, or accuracy, or noise robustness. Simulating a tiny portion of it in SPICE does not really provide much insight. When the chip is built, and working, then we can compare it to the best analog "continuous rate" chip running the same model, and only then we will be able to see which one is more efficient. Until that time, any claims that spikes are more efficient are unsubstantiated. On the other hand, if you can devise an algorithm which is uniquely suited for spiking networks (biologically plausible backprop, or whatever), then sure, it's quite possible that you will be able to do things more efficiently. So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? That's not a rhetorical question, perhaps there are good reasons, and I'd like to know them.

Seanny123 10y ago
That's totally fair to want to wait for a comparison until there's actual hardware produced. Especially for comparisons exclusively involving DL models. Make initial "area under the curve" argument was rhetorical and not sufficiently empirically founded. > If they are indeed comparing SNN to a GPU in that paper, then it's just plain horrible! Yeah, the paper talks about how this is preliminary and bigger savings are expected for video. However, I must concede as before that without the analog hardware built there isn't much point discussing this. > So, my question is, why try mapping DL models, trained on "traditional hardware" to SNNs, which weren't designed to run them? Why not focus instead on finding those biologically plausible algorithms first? If your goal is to understand the brain, wouldn't it be more reasonable to continue experiment in software until you do? Why build hardware to understand brain? Eric Hunsberger [1] has been doing most of the work in this domain, so I'm going to be awkwardly paraphrasing my conversations with him. Eric wanted to make Spaun's [2] vision system better. To do that, he knew he was going to need ConvNets or at least build something off of them. So he started to see if he could bring ConvNets into the domain of SNNs to understand them better. Once he did that, he started looking into if he could learn the SNN ConvNets using biologically plausible back-prop [3] which is where he's at now. That's really only a branch of our research. To understand the brain, we have a lot of really different methods that are more based in Symbolicism, Bayesianism and Dynamicism. We do start in software [2], but software is slow, even on a GPU. When we get faster hardware, we're able to explore the algos more quickly. Also, we got funding to build/explore analog hardware from the Office of Naval Research, so that's where this project is investigating. To summarize, the DNN-to-SNN-adaptation algos aren't the only thing targeting this hardware, but a small slice of a family of algos that are defining the requirements of the hardware. (I hope this post confirms that I understood and accept your argument and isn't me having a case of "must have the last word"-ism) [1] http://compneuro.uwaterloo.ca/people/eric-hunsberger.html http://compneuro.uwaterloo.ca/people/eric-hunsberger.html [2] https://nengo.github.io/ https://nengo.github.io/ [3] http://cogsci.stackexchange.com/q/16269/4397 http://cogsci.stackexchange.com/q/16269/4397
- p1esk 10y ago
  I appreciate your response, especially the backstory of running a convnet on SNN hardware. A couple of remarks: 1. Your own response to your stackexchange question: "The random synaptic feedback weights only accomplish back-propagation through one layer of neurons, thus severely limiting the depth of a network." Didn't they show in the paper how it could work through multiple layers? 2. software is slow, even on a GPU. When we get faster hardware, we're able to explore the algos more quickly I'm not sure what you're referring to by "faster hardware", but somehow I doubt you will beat a workstation with 8 GPUs in terms of speed, if your goal is to explore algos more quickly. More importantly, what if the next algo you want to explore does not map well to the hardware you built? For example, what if we realize that relative timings between spikes are important, and most of the computation is based on that, but your hardware was not designed to exploit this "race logic" principles? Suddenly your custom system became much less useful, while your GPUs will simulate that just fine. There's a possibility that instead of exploring the best or most plausible algorithms, you will limit yourself to algorithms which map well to your hardware. 3. What do you think about HTM theory by Numenta? They strive for biologically realistic computation, but they don't think spikes are important, and abstract them away in their code. p.s. the reason I'm involved in this discussion is I'm trying to decide whether to accept an internship offer to work on Bayesian inference algorithms for SNN chip.