4 ms·

Does anyone know how these things actually learn? I saw something in the article like "data passes over the network and creates reinforcement" but how well does

Does anyone know how these things actually learn? I saw something in the article like "data passes over the network and creates reinforcement" but how well does this work, and how?

samhain 8y ago
That's a really good question... I know that normal network training is basically "y=mx+b, solve for m and b." But then you add layers and transfer functions, so it's more like: y1=f(m1y2)+b1 y2=f(m2yn)+b2 ... And then you solve for each mn and bn, using f^(-1), which is why smooth transfer functions are preferred, and then on some networks you can visualize the training space, with the derivative pointing toward the most optimal position. But, a spiking neural network that isn't smooth seems like you wouldn't form clean gradients for training, so that's actually a really good question... Seems like it wouldn't work correctly, or would be incredibly difficult to train. Of course maybe "spiking" is a name for another part of the behavior, and not the transfer function itself.
accurrent 8y ago
A common algorithm used in Spiking Neural Networks is called STDP (Stochastic Timing Dependent Plasticity). I do not know what exactly brainchip uses though. It is also possible to gradient descent your way through some SNNs although this is very inefficient. Also it is possible to directly convert many CNN architectures to equivalent spiking architectures. (i.e. Learn a CNN using gradient descent then approximate it as an SNN).
- burning_hamster 8y ago
  FYI: STDP generally stands for spike time dependent plasticity.
buboard 8y ago
These architectures are hardware implementations of simple integrate-and-fire-related architectures. You can usually implement simple plasticity algorithms like spike-timing dependent plasticity. They don't exceed ANNs at this point, but are closer to biological neurons. Some researchers are looking into how the backpropagation of errors might be implemented in spiking networks. Example: https://www.cell.com/neuron/abstract/S0896-6273(13)01127-6?code=cell-site https://www.cell.com/neuron/abstract/S0896-6273(13)01127-6?c...

Does anyone know how these things actually learn? I saw something in the article like "data passes over the network and creates reinforcement" but how well does this work, and how?

samhain 8y ago

That's a really good question... I know that normal network training is basically "y=mx+b, solve for m and b." But then you add layers and transfer functions, so it's more like: y1=f(m1y2)+b1 y2=f(m2yn)+b2 ... And then you solve for each mn and bn, using f^(-1), which is why smooth transfer functions are preferred, and then on some networks you can visualize the training space, with the derivative pointing toward the most optimal position. But, a spiking neural network that isn't smooth seems like you wouldn't form clean gradients for training, so that's actually a really good question... Seems like it wouldn't work correctly, or would be incredibly difficult to train. Of course maybe "spiking" is a name for another part of the behavior, and not the transfer function itself.

accurrent 8y ago

A common algorithm used in Spiking Neural Networks is called STDP (Stochastic Timing Dependent Plasticity). I do not know what exactly brainchip uses though. It is also possible to gradient descent your way through some SNNs although this is very inefficient. Also it is possible to directly convert many CNN architectures to equivalent spiking architectures. (i.e. Learn a CNN using gradient descent then approximate it as an SNN).

burning_hamster 8y ago
FYI: STDP generally stands for spike time dependent plasticity.

buboard 8y ago

These architectures are hardware implementations of simple integrate-and-fire-related architectures. You can usually implement simple plasticity algorithms like spike-timing dependent plasticity. They don't exceed ANNs at this point, but are closer to biological neurons. Some researchers are looking into how the backpropagation of errors might be implemented in spiking networks. Example: https://www.cell.com/neuron/abstract/S0896-6273(13)01127-6?code=cell-site https://www.cell.com/neuron/abstract/S0896-6273(13)01127-6?c...