3 ms·

Why do these articles on RNNs and "Deep Learning" never mention Hierarchical Temporal Memory? https://en.wikipedia.org/wiki/Hierarchical_temporal_memory#Deep_L

Why do these articles on RNNs and "Deep Learning" never mention Hierarchical Temporal Memory? https://en.wikipedia.org/wiki/Hierarchical_temporal_memory#Deep_Learning https://en.wikipedia.org/wiki/Hierarchical_temporal_memory#D...

akyu 11y ago
It seems like theres a kind of taboo against HTM in the ML community. I guess it stems from their lack of impressive benchmarks, but I think that's kind of missing the point when it comes to HTM. Maybe HTM isn't the right solution, but I think there is a lot to be learned by using neocortex inspired models, and HTM is at least a solid step in that direction. And the work Numenta has contributed on Sparse Codings shouldn't be overlooked.
- davmre 11y ago
  I don't think most ML people are actively hostile to HTM, just indifferent until they show some results. For example, Yann Lecun in his Reddit AMA: http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_yann_lecun/chisjsc http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_...
- ajays 11y ago
  I've yet to find one significant benchmark dataset where HTM beats other methods. One.
- dwf 11y ago
  Extraordinary claims demand extraordinary evidence. Numenta has plenty of the former and nil as regards the latter. Even if they had invented some believably interesting task and done fair comparisons with other methods and shown that HTM succeeds where others fail, it would be considered worth a look by the wider machine learning community.
Teodolfo 11y ago
Because it doesn't meet academic standards for publishing in NIPS and ICML, the most prestigious machine learning conferences. Edit: To clarify, research papers generally cite other peer-reviewed research papers in similar venues preferentially. ML papers should mostly be citing ML papers in high-quality, peer-reviewed venues. HTM doesn't have papers like this to cite.
bra-ket 11y ago
These communities have different goals from the very beginning. Pattern recognition vs. real intelligence, which is what HTM is about. Hawkins describes this gap well in his book. But there is some cross-pollination, see the recent projects by Stan Franklin's lab on Sparse distributed memory and composite representations, it's a step towards integration with deep learning: http://ccrg.cs.memphis.edu/assets/papers/theses-dissertations/Snaider_dissertation.pdf http://ccrg.cs.memphis.edu/assets/papers/theses-dissertation... On the other hand check out the work by Volodymyr Mnih from DeepMind https://www.cs.toronto.edu/~vmnih/ https://www.cs.toronto.edu/~vmnih/, reinforcement learning with "visual attention" is a step towards consciousness models of the HTM/SDM/LIDA camp.
- paulsutter 11y ago
  I was also under the misapprehension that deep learning is just about classification, but that isn't true. Yes, reinforcement learning is the path to general intelligence, and the deep learning community is showing impressive progress on that front as well. The Deepmind demo [1] and the recent robotics work at Berkeley[2] are good examples. Thanks for the link to Stan Franklin's work. I'm glad to hear there is work to integrate the two approaches. [1] https://www.youtube.com/watch?v=EfGD2qveGdQ https://www.youtube.com/watch?v=EfGD2qveGdQ [2] http://newscenter.berkeley.edu/2015/05/21/deep-learning-robot-masters-skills-via-trial-and-error/ http://newscenter.berkeley.edu/2015/05/21/deep-learning-robo...
paulsutter 11y ago
Jeff Hawkins takes the position that only Numenta sees the human brain as a temporal prediction engine based on sparse, hierarchical memory. But actually, RNNs are great for recognizing and predicting temporal sequences (as we saw in the Karpathy post), RNNs use a sparse representation, and RNNs can be extended with hierarchical memory [1] The big difference is that the neural network crowd are getting some spectacular results, and Numenta, well, maybe they'll be show more progress in the future. Jeff Hawkins is super smart and a good guy, and he might get more done if he acknowledged the commonalities in the approaches rather than having to invent it all separately at Numenta. I really dont mean to be critical. Jeff inspired my own interest in machine intelligence. [1] page 442, "Over the past year, several authors have made different proposals to augment RNNs with a memory module. Proposals include the Neural Turing Machine in which the network is augmented by a ‘tape-like’ memory that the RNN can choose to read from or write to, and memory networks, in which a regular network is augmented by a kind of associative memory. Memory networks have yielded excellent performance on standard question-answering benchmarks. The memory is used to remember the story about which the network is later asked to answer questions."

akyu 11y ago

It seems like theres a kind of taboo against HTM in the ML community. I guess it stems from their lack of impressive benchmarks, but I think that's kind of missing the point when it comes to HTM. Maybe HTM isn't the right solution, but I think there is a lot to be learned by using neocortex inspired models, and HTM is at least a solid step in that direction. And the work Numenta has contributed on Sparse Codings shouldn't be overlooked.

davmre 11y ago
I don't think most ML people are actively hostile to HTM, just indifferent until they show some results. For example, Yann Lecun in his Reddit AMA: http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_yann_lecun/chisjsc http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_...
ajays 11y ago
I've yet to find one significant benchmark dataset where HTM beats other methods. One.
dwf 11y ago
Extraordinary claims demand extraordinary evidence. Numenta has plenty of the former and nil as regards the latter. Even if they had invented some believably interesting task and done fair comparisons with other methods and shown that HTM succeeds where others fail, it would be considered worth a look by the wider machine learning community.

Teodolfo 11y ago

Because it doesn't meet academic standards for publishing in NIPS and ICML, the most prestigious machine learning conferences. Edit: To clarify, research papers generally cite other peer-reviewed research papers in similar venues preferentially. ML papers should mostly be citing ML papers in high-quality, peer-reviewed venues. HTM doesn't have papers like this to cite.

bra-ket 11y ago

These communities have different goals from the very beginning. Pattern recognition vs. real intelligence, which is what HTM is about. Hawkins describes this gap well in his book. But there is some cross-pollination, see the recent projects by Stan Franklin's lab on Sparse distributed memory and composite representations, it's a step towards integration with deep learning: http://ccrg.cs.memphis.edu/assets/papers/theses-dissertations/Snaider_dissertation.pdf http://ccrg.cs.memphis.edu/assets/papers/theses-dissertation... On the other hand check out the work by Volodymyr Mnih from DeepMind https://www.cs.toronto.edu/~vmnih/ https://www.cs.toronto.edu/~vmnih/, reinforcement learning with "visual attention" is a step towards consciousness models of the HTM/SDM/LIDA camp.

paulsutter 11y ago
I was also under the misapprehension that deep learning is just about classification, but that isn't true. Yes, reinforcement learning is the path to general intelligence, and the deep learning community is showing impressive progress on that front as well. The Deepmind demo [1] and the recent robotics work at Berkeley[2] are good examples. Thanks for the link to Stan Franklin's work. I'm glad to hear there is work to integrate the two approaches. [1] https://www.youtube.com/watch?v=EfGD2qveGdQ https://www.youtube.com/watch?v=EfGD2qveGdQ [2] http://newscenter.berkeley.edu/2015/05/21/deep-learning-robot-masters-skills-via-trial-and-error/ http://newscenter.berkeley.edu/2015/05/21/deep-learning-robo...

paulsutter 11y ago

Jeff Hawkins takes the position that only Numenta sees the human brain as a temporal prediction engine based on sparse, hierarchical memory. But actually, RNNs are great for recognizing and predicting temporal sequences (as we saw in the Karpathy post), RNNs use a sparse representation, and RNNs can be extended with hierarchical memory [1] The big difference is that the neural network crowd are getting some spectacular results, and Numenta, well, maybe they'll be show more progress in the future. Jeff Hawkins is super smart and a good guy, and he might get more done if he acknowledged the commonalities in the approaches rather than having to invent it all separately at Numenta. I really dont mean to be critical. Jeff inspired my own interest in machine intelligence. [1] page 442, "Over the past year, several authors have made different proposals to augment RNNs with a memory module. Proposals include the Neural Turing Machine in which the network is augmented by a ‘tape-like’ memory that the RNN can choose to read from or write to, and memory networks, in which a regular network is augmented by a kind of associative memory. Memory networks have yielded excellent performance on standard question-answering benchmarks. The memory is used to remember the story about which the network is later asked to answer questions."