3 ms·

Spiking neural networks do have many advantages over conventional rate based networks. They can, for example, handle time which a convnet cannot do without reso

Spiking neural networks do have many advantages over conventional rate based networks. They can, for example, handle time which a convnet cannot do without resorting to frames. They also have a lot of disadvantages, the main one being that no one has any idea of how to use them properly yet.

deepnotderp 10y ago
RNNs can handle the idea of time.
- skummetmaelk 10y ago
  Not very well though. You still need to operate on frames because you need a time interval for capturing the firing rate that conventional networks model. As you make the interval smaller, which you need for quick reaction times, you make the network more noisy and unstable. SNNs can work in real time.
  - p1esk 10y ago
    Are you implying that spiking networks are somehow faster than level based ones? We can build analog level based networks and I don't see any reason why they would have to be slower than spike based ones. And as far as the noise, it's also unclear to me why a spiking network would be more noise resistant or stable. Do tell.
    - skummetmaelk 10y ago
      The prevailing theory for why the brain uses spiking networks is that it is much more energy efficient to transmit "all-or-nothing" spikes versus propagating analog membrane potential across very large axonal trees. Contemporary machine learning is focused on rate-based neural networks with continuous activation levels representing the firing rate of the neurons. This makes these networks unable to measure the time difference between spikes arriving at a single neuron, an ability which is known to play a critical role in your auditory system for example. It is unknown how it is used in other parts of your brain. However, in recent years the concept of dendritic computation, defined as computation being done by the complex interaction of synaptic activations as they pass down a dendritic tree before reaching the neuron soma, has gained a lot of attention. If you do not create networks where time is a "first-class citizen" you will be totally unable to do this. Of course this might be more of an academic interest depending on your point of view. The power efficiency argument from biology translates well to silicon when you want to build large structures. Here you cannot have 32-bit floats being passed around to thousands of target synapses, so you must make do with less.
      - p1esk 10y ago
        The way brain computes is shaped by biological constraints. It's true that using spikes in a brain is very power efficient, however, we are discussing building computation systems in silicon, and I don't understand why people believe using spikes in silicon is more power efficient than using analog voltage levels (or currents). I'm not sure what you tried to say about 32 bit floats: are you saying it's easier or more efficient to represent a 32 bit float using time difference between spikes than using a voltage level? it might be true, but I don't see how.
        skummetmaelk 10y ago
        It is definitely easier to transmit spikes across a silicon chip than analog voltages or digital numbers. Analog voltages will be subject to voltage drops as they traverse the chip and require a massive number of wires because you cannot multiplex the voltages without extremely expensive (area wise) analog multiplexers. Transmitting digital numbers likewise require a larger number of wires.
        p1esk 10y ago
        It is definitely easier to transmit spikes across a silicon chip than analog voltages Why? We are not trying to transmit spikes, we are trying to transmit information. It's not at all clear to me that encoding a signal as a series of spikes is somehow more efficient than placing a certain voltage level on a wire. More specifically, given a particular noise floor, and a desired data rate, you either use binary encoding (spikes), but have to use higher frequency, or you use lower frequency but pack more bits into each voltage level. This would probably be affected a lot by the required frequency range. One example comes to mind: T1 transmission lines (NRZ encoding), and DSL phone lines (QAM). I don't know which one is more power efficient, and not sure if that was even a concern there, but it would be interesting to know how they compare in terms of power efficiency, and how that changes when we increase data rates or noise levels. Analog voltages will be subject to voltage drops as they traverse the chip The spikes will be subject to exactly the same voltage drops, what's your point? and require a massive number of wires because you cannot multiplex the voltages without extremely expensive (area wise) analog multiplexers I'm not sure what you mean here. What do you want to multiplex, and why spikes are better for that?
jhj 10y ago
ConvNets can handle sequences just as well (cf. TDNNs), you just treat time as another dimension. Whether you want to call this a "frame" or not is subject to question. https://arxiv.org/pdf/1611.02344.pdf https://arxiv.org/pdf/1611.02344.pdf With an explicit (potentially arbitrarily large) memory component, they can also capture long-term dependencies that don't have to be maintained in, say, the convolution parameters between presentations of data. This is one thing we want to pursue with the Faiss library at FAIR. Regardless of how something is implemented, it's the what that is important. What's the actual algorithm or algorithms that the brain uses? It's likely easier to explore this question in floating-point math than putting together a lot of spiking neuron like things and trying to cargo cult your way to the answer.
- emcq 10y ago
  It's interesting you used a Feynman reference here, as he was actually involved spawning the neuromorphic field at Caltech with Carver Mead [0]. You're thinking about this as if it's purely an algorithm problem and that computer architecture should always be designed to do the algorithms bidding. Neuromorphic is flipping the problem around and creates efficient architecture where algorithms do the architectures bidding. The former is much better in the cloud with massive armies of general purpose computers while the latter is much better on the edge for anything implantable (very little heat can be generated or else you cook) or needs super low power (less than 1W). Having worked on both sides of the spectrum I think there's room for both and the relative research funding seems reasonable; much more invested in straightforward neural network research with GPUs etc versus the neuromorphic field where it's mostly a handful of Caltech folks left at places like Stanford, UCSD, GT, John's Hopkins, or UF. [0] https://en.m.wikipedia.org/wiki/Carver_Mead https://en.m.wikipedia.org/wiki/Carver_Mead

deepnotderp 10y ago

RNNs can handle the idea of time.

skummetmaelk 10y ago
Not very well though. You still need to operate on frames because you need a time interval for capturing the firing rate that conventional networks model. As you make the interval smaller, which you need for quick reaction times, you make the network more noisy and unstable. SNNs can work in real time.
- p1esk 10y ago
  Are you implying that spiking networks are somehow faster than level based ones? We can build analog level based networks and I don't see any reason why they would have to be slower than spike based ones. And as far as the noise, it's also unclear to me why a spiking network would be more noise resistant or stable. Do tell.
  - skummetmaelk 10y ago
    The prevailing theory for why the brain uses spiking networks is that it is much more energy efficient to transmit "all-or-nothing" spikes versus propagating analog membrane potential across very large axonal trees. Contemporary machine learning is focused on rate-based neural networks with continuous activation levels representing the firing rate of the neurons. This makes these networks unable to measure the time difference between spikes arriving at a single neuron, an ability which is known to play a critical role in your auditory system for example. It is unknown how it is used in other parts of your brain. However, in recent years the concept of dendritic computation, defined as computation being done by the complex interaction of synaptic activations as they pass down a dendritic tree before reaching the neuron soma, has gained a lot of attention. If you do not create networks where time is a "first-class citizen" you will be totally unable to do this. Of course this might be more of an academic interest depending on your point of view. The power efficiency argument from biology translates well to silicon when you want to build large structures. Here you cannot have 32-bit floats being passed around to thousands of target synapses, so you must make do with less.
    - p1esk 10y ago
      The way brain computes is shaped by biological constraints. It's true that using spikes in a brain is very power efficient, however, we are discussing building computation systems in silicon, and I don't understand why people believe using spikes in silicon is more power efficient than using analog voltage levels (or currents). I'm not sure what you tried to say about 32 bit floats: are you saying it's easier or more efficient to represent a 32 bit float using time difference between spikes than using a voltage level? it might be true, but I don't see how.
      - skummetmaelk 10y ago
        It is definitely easier to transmit spikes across a silicon chip than analog voltages or digital numbers. Analog voltages will be subject to voltage drops as they traverse the chip and require a massive number of wires because you cannot multiplex the voltages without extremely expensive (area wise) analog multiplexers. Transmitting digital numbers likewise require a larger number of wires.
        p1esk 10y ago
        It is definitely easier to transmit spikes across a silicon chip than analog voltages Why? We are not trying to transmit spikes, we are trying to transmit information. It's not at all clear to me that encoding a signal as a series of spikes is somehow more efficient than placing a certain voltage level on a wire. More specifically, given a particular noise floor, and a desired data rate, you either use binary encoding (spikes), but have to use higher frequency, or you use lower frequency but pack more bits into each voltage level. This would probably be affected a lot by the required frequency range. One example comes to mind: T1 transmission lines (NRZ encoding), and DSL phone lines (QAM). I don't know which one is more power efficient, and not sure if that was even a concern there, but it would be interesting to know how they compare in terms of power efficiency, and how that changes when we increase data rates or noise levels. Analog voltages will be subject to voltage drops as they traverse the chip The spikes will be subject to exactly the same voltage drops, what's your point? and require a massive number of wires because you cannot multiplex the voltages without extremely expensive (area wise) analog multiplexers I'm not sure what you mean here. What do you want to multiplex, and why spikes are better for that?

jhj 10y ago

ConvNets can handle sequences just as well (cf. TDNNs), you just treat time as another dimension. Whether you want to call this a "frame" or not is subject to question. https://arxiv.org/pdf/1611.02344.pdf https://arxiv.org/pdf/1611.02344.pdf With an explicit (potentially arbitrarily large) memory component, they can also capture long-term dependencies that don't have to be maintained in, say, the convolution parameters between presentations of data. This is one thing we want to pursue with the Faiss library at FAIR. Regardless of how something is implemented, it's the what that is important. What's the actual algorithm or algorithms that the brain uses? It's likely easier to explore this question in floating-point math than putting together a lot of spiking neuron like things and trying to cargo cult your way to the answer.

emcq 10y ago
It's interesting you used a Feynman reference here, as he was actually involved spawning the neuromorphic field at Caltech with Carver Mead [0]. You're thinking about this as if it's purely an algorithm problem and that computer architecture should always be designed to do the algorithms bidding. Neuromorphic is flipping the problem around and creates efficient architecture where algorithms do the architectures bidding. The former is much better in the cloud with massive armies of general purpose computers while the latter is much better on the edge for anything implantable (very little heat can be generated or else you cook) or needs super low power (less than 1W). Having worked on both sides of the spectrum I think there's room for both and the relative research funding seems reasonable; much more invested in straightforward neural network research with GPUs etc versus the neuromorphic field where it's mostly a handful of Caltech folks left at places like Stanford, UCSD, GT, John's Hopkins, or UF. [0] https://en.m.wikipedia.org/wiki/Carver_Mead https://en.m.wikipedia.org/wiki/Carver_Mead