4 ms·

I am glad he pointed out that most artificial neural networks bear only a superficial resemblance to our own biological ones. But I think he failed to appreciat

by jostmey 12y ago

I am glad he pointed out that most artificial neural networks bear only a superficial resemblance to our own biological ones. But I think he failed to appreciate the power behind Boltzmann Machines - a type of neural network designed to create a generative model of a dataset. Personally, I find the resemblance between these neural networks and the real ones a little uncanny. And very few people seem to realize that the formalism behind a Boltzmann Machine can be adapted to fit the activation patterns of real neurons - you just have to redefine the energy function to match that of real biological neurons.

scottlocklin 12y ago
I've only started looking at RBMs recently, but ... what are you talking about? Biological neural networks use spikes. RBMs certainly don't. RBMs look more like HMMs to me than like biological neurons. Don't take this as me saying, "you're wrong" -I'm curious if there is another way to think about RBMs (aka, "papers please" -so I have a deeper understanding when I do my own implementation of RBMs).
- jostmey 12y ago
  The energy function for a Boltzmann machine is usually E = v^T x W x h, where v are the visible units, W contains the weights, and h represents the hidden units. The form of the energy function defines the activation function of each neuron and the learning rule that goes with it. Now, you can in theory start off with any energy function you like (this form just happens to be the simplest). You would then have to re-derive the activation function and learning rule. Just what would the energy function look like for real neurons? I don't know but we do know that the activation function would have to "spike" in bursts. So that is a clue. We also have rudimentary ideas about the learning rule used in biological neural networks, so you would also want to take this into account when determining the actual energy function. Finally, real neurons do not send retrograde signals but are instead wired recurrently, which must also be taken into consideration.

by jostmey 12y ago

scottlocklin 12y ago

I've only started looking at RBMs recently, but ... what are you talking about? Biological neural networks use spikes. RBMs certainly don't. RBMs look more like HMMs to me than like biological neurons. Don't take this as me saying, "you're wrong" -I'm curious if there is another way to think about RBMs (aka, "papers please" -so I have a deeper understanding when I do my own implementation of RBMs).

jostmey 12y ago
The energy function for a Boltzmann machine is usually E = v^T x W x h, where v are the visible units, W contains the weights, and h represents the hidden units. The form of the energy function defines the activation function of each neuron and the learning rule that goes with it. Now, you can in theory start off with any energy function you like (this form just happens to be the simplest). You would then have to re-derive the activation function and learning rule. Just what would the energy function look like for real neurons? I don't know but we do know that the activation function would have to "spike" in bursts. So that is a clue. We also have rudimentary ideas about the learning rule used in biological neural networks, so you would also want to take this into account when determining the actual energy function. Finally, real neurons do not send retrograde signals but are instead wired recurrently, which must also be taken into consideration.