5 ms·

I wouldn't be so sure that Boltzmann machines were designed without how the brain operated in mind. Terry Seknowski, one of the inventors, is a computational ne

I wouldn't be so sure that Boltzmann machines were designed without how the brain operated in mind. Terry Seknowski, one of the inventors, is a computational neuroscientist. In general, I think the influence of neuroscience on neural network research has been subtle and perhaps underrated. For example, modern day convolutional neural networks Havea lineage going back to Fukushima's neocognitron, which was heavily inspired by Hubel an Wiesel's simple/complex cell model of visual cortex based on single cell electrophysiology.

tprice7 10y ago
"I wouldn't be so sure that Boltzmann machines were designed without how the brain operated in mind." Sure, actually it seems highly unlikely to me that they were designed without considering at all how they brain works. Let me clarify what I meant: given the definition of how a Boltzmann machine infers (i.e. how to determine which neurons are on or off), the training algorithm can be derived from purely mathematical considerations.
- jayajay 10y ago
  When I told Hinton about SPWs and that they are related to RBMs, he told me that he had a theory which required the existence of SPWs. That is not a coincidence, they seem to have been inspired by neuroscience and real neural network phenomena.
  - giardini 10y ago
    What's an SPW?
    - jayajay 10y ago
      SPW: High frequency oscillation in the hippocampus which occurs during rest and sleep. The characteristics would suggest they are critical in memory management. If you google "SPW neuroscience", you will find many resources. https://en.wikipedia.org/wiki/Sharp_waves_and_ripples https://en.wikipedia.org/wiki/Sharp_waves_and_ripples Brain science is highly interdisciplinary. Each expert should spend time to familiarize themselves with the various approaches used by other experts. In particular, machine learning theorists should spend time to understand the latest results (their implications, at least) in neuroscience and neuron biophysics. At some point, an ML theorist is not too different than a biophysics theorist. They just use different abstractions. This is why I am particularly interested in Google's and Hinton's work. That work seems to be subtly motivated by neuroscience and natural science...
mlechha 10y ago
They weren't. They were a generalization of the Hopfield networks. Boltzmann machines are a stochastic version of the Hopfield network. The training algorithm simply tries to minimize the KL divergence between the network activity and real data. So it was quite surprising when it turned out that the algorithm needed a "dream phase" as they call it. Francis Crick was inspired by this and proposed a theory of sleep.
- redler 10y ago
  Unpacking a comment like this is one of the quieter pleasures of reading HN.
  - vkreso 10y ago
    Couldn't agree more
  - mlechha 10y ago
    Haha I'm not sure if you're being sarcastic so I'll try to unpack the comment. Hopfield networks were one of the first models of associative memory. They themselves were based on a model of simple magnets called ising model (generalized). Basically a group of binary units, each connected its nearest neighbors with a coupling strength. Each unit prefers to be like their neighbors. Hopfield developed a clever method to change the coupling so that the networks can store and retrieve patterns of activity. In the Hopfield network everything is deterministic, Hopfield himself realized that if this constraint was relaxed this model could become a very powerful computational machine. Which means that if instead of being always on or off, the units had a probability of being on or off the networks could perform very general computational tools [1]. Unfortunately, training these general stochastic systems was not easy. With their Boltzmann machines Sejnowski and Hinton proposed a possible solution. The activity of stochastic binary units effectively encodes a probability distribution, so all they had to do was make sure that the probability distribution being encoded by the activity of the units was the same as that of the input. They did this by changing the connection strengths between the units such that the activity pattern minimized something called the Kullback-Leibler or KL divergence, which is a measure of how close two probability distributions are (the one encoded by the network activity, or the dream activity of the network and the probability distribution of the real data e.g. a set of natural images). If two distributions match exactly then the KLd is zero and if not it's large. When they wrote out the math it turned out that the algorithm required two phases, an awake phase where the connections were changed according to the real data, and the sleep phase where the connections were pruned by the spontaneous activity of the network without any input (or dreams). This analogy got a lot of people excited, including Francis Crick and several others tried to test this idea in real brains, but we are still waiting for a convincing result.
espeed 10y ago
Numenta [1] (co-founded by Jeff Hawkins, author of "On Intelligence") has been working on a model of ANN designed around how the brain operates. Hierarchical temporal memory (HTM) [2] is one aspect of their model. 1. http://numenta.com http://numenta.com 2. https://en.wikipedia.org/wiki/Hierarchical_temporal_memory https://en.wikipedia.org/wiki/Hierarchical_temporal_memory

tprice7 10y ago

"I wouldn't be so sure that Boltzmann machines were designed without how the brain operated in mind." Sure, actually it seems highly unlikely to me that they were designed without considering at all how they brain works. Let me clarify what I meant: given the definition of how a Boltzmann machine infers (i.e. how to determine which neurons are on or off), the training algorithm can be derived from purely mathematical considerations.

jayajay 10y ago
When I told Hinton about SPWs and that they are related to RBMs, he told me that he had a theory which required the existence of SPWs. That is not a coincidence, they seem to have been inspired by neuroscience and real neural network phenomena.
- giardini 10y ago
  What's an SPW?
  - jayajay 10y ago
    SPW: High frequency oscillation in the hippocampus which occurs during rest and sleep. The characteristics would suggest they are critical in memory management. If you google "SPW neuroscience", you will find many resources. https://en.wikipedia.org/wiki/Sharp_waves_and_ripples https://en.wikipedia.org/wiki/Sharp_waves_and_ripples Brain science is highly interdisciplinary. Each expert should spend time to familiarize themselves with the various approaches used by other experts. In particular, machine learning theorists should spend time to understand the latest results (their implications, at least) in neuroscience and neuron biophysics. At some point, an ML theorist is not too different than a biophysics theorist. They just use different abstractions. This is why I am particularly interested in Google's and Hinton's work. That work seems to be subtly motivated by neuroscience and natural science...

mlechha 10y ago

They weren't. They were a generalization of the Hopfield networks. Boltzmann machines are a stochastic version of the Hopfield network. The training algorithm simply tries to minimize the KL divergence between the network activity and real data. So it was quite surprising when it turned out that the algorithm needed a "dream phase" as they call it. Francis Crick was inspired by this and proposed a theory of sleep.

redler 10y ago
Unpacking a comment like this is one of the quieter pleasures of reading HN.
- vkreso 10y ago
  Couldn't agree more
- mlechha 10y ago
  Haha I'm not sure if you're being sarcastic so I'll try to unpack the comment. Hopfield networks were one of the first models of associative memory. They themselves were based on a model of simple magnets called ising model (generalized). Basically a group of binary units, each connected its nearest neighbors with a coupling strength. Each unit prefers to be like their neighbors. Hopfield developed a clever method to change the coupling so that the networks can store and retrieve patterns of activity. In the Hopfield network everything is deterministic, Hopfield himself realized that if this constraint was relaxed this model could become a very powerful computational machine. Which means that if instead of being always on or off, the units had a probability of being on or off the networks could perform very general computational tools [1]. Unfortunately, training these general stochastic systems was not easy. With their Boltzmann machines Sejnowski and Hinton proposed a possible solution. The activity of stochastic binary units effectively encodes a probability distribution, so all they had to do was make sure that the probability distribution being encoded by the activity of the units was the same as that of the input. They did this by changing the connection strengths between the units such that the activity pattern minimized something called the Kullback-Leibler or KL divergence, which is a measure of how close two probability distributions are (the one encoded by the network activity, or the dream activity of the network and the probability distribution of the real data e.g. a set of natural images). If two distributions match exactly then the KLd is zero and if not it's large. When they wrote out the math it turned out that the algorithm required two phases, an awake phase where the connections were changed according to the real data, and the sleep phase where the connections were pruned by the spontaneous activity of the network without any input (or dreams). This analogy got a lot of people excited, including Francis Crick and several others tried to test this idea in real brains, but we are still waiting for a convincing result.

espeed 10y ago

Numenta [1] (co-founded by Jeff Hawkins, author of "On Intelligence") has been working on a model of ANN designed around how the brain operates. Hierarchical temporal memory (HTM) [2] is one aspect of their model. 1. http://numenta.com http://numenta.com 2. https://en.wikipedia.org/wiki/Hierarchical_temporal_memory https://en.wikipedia.org/wiki/Hierarchical_temporal_memory