3 ms·

Hawkins' mantra is sequence based memory. I love this guy and his work. And although I think his company has had success with sequence based AI, it still doe

by mjpuser 11y ago

Hawkins' mantra is sequence based memory. I love this guy and his work. And although I think his company has had success with sequence based AI, it still doesn't totally answer how the brain works. His model basically works by taking encoded symbols and being able to predict the next encoded symbol. This works great for things with patterns (power consumption, traffic patterns, etc), but it doesn't work with language. You will not be able to predict what I am going to say, or predict my answer based on the question, at least not very well. The problem is that he needs to figure out meta computation. His product, nupic, would take in a string of words, which gets encoded into a semantic representation, and tries to predict what's next. But concepts are larger than a single word, and until it can build a working semantic representation of what it's currently processing, it will not prove to be a useful model in describing how the brain works, or at least being able to compare it to our brain.

mietek 11y ago
> This works great for things with patterns, but it doesn't work with language. Sure it does. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ > Hawkins' mantra is sequence based memory. Note that Hawkins wrote “On intelligence” in 2005. Schmidhuber wrote “Learning complex extended sequences using the principle of history compression” in 1992, and “Long short-term memory”, with Hochreiter, in 1997. Today, a LSTM-based RNN can answer your Gmail. http://people.idsia.ch/~juergen/rnn.html http://people.idsia.ch/~juergen/rnn.html http://googleresearch.blogspot.com/2015/11/computer-respond-to-this-email.html http://googleresearch.blogspot.com/2015/11/computer-respond-...
- mjpuser 11y ago
  Sorry! To clarify that it doesn't work with language, I mean you can't ask it a question and then expect it to come up with an answer. For most meaningful questions, you can't sequence an answer from it's question in a general way. This might be in part because Hawkins isn't trying to mimic the brain, but learn about how it works to apply the algorithms it uses to our needs/wants.
  - p1esk 11y ago
    you can't ask it a question and then expect it to come up with an answer Why not? If the model was trained on a large enough corpus of texts, it will have seen lots of answers to your question, or similar questions. It can, in principle, extract the important features of those answers, and present them to you as an answer. This is pretty much how the majority of humans would answer "the most meaningful" questions.
    - mjpuser 11y ago
      Nupic showcases it's "What did the fox eat" use case which uses cortical.io's SDRs of words. In this example it "asks the question" what does the fox eat after teach it what other animals eat, and without ever seeing fox before, it's able to accurately say what it eats since it groups foxes with coyotes, etc, which have a similar SDR. However, this is a sequence... They fed it sentences only in the form "a eats b" and then fed it "fox eats", and it replied "rodent" or whatever... The SDR's have to be sequences in order for the HTM to work. So if you ask "Guess how many balls are in this jug", you have to do an implied computation to guess how many possible balls fit inside the jug, and that implied computation is not the word sequence (or sdr sequence). Even if you gave it an infinite amount of similar questions and answers like this, it would never be able to figure out a general way to answer the question, which means you could always stump it by giving a different sized jug + different sized ball.
      - p1esk 11y ago
        H in HTM stands for "Hierarchical". This means, that it can, in principle, construct more general, more abstract SDRs (patterns) based on many low level SDRs it sees. Thus it's plausible to see forming of ideas out of sentences, or something similar. This is how it can build a world model, and then run your input through multiple levels of abstraction, as many as needed to give a high confidence answer.
        mjpuser 11y ago
        My example was also spoon feeding questions and answers to the HTM. If you were to just give it a corpus of text, and then ask it a question about that text, it would not be able to formulate an answer that has any meaning, and follow proper English grammar. If you disagree, you can respond with a working example.
RobertoG 11y ago
If you study the Numenta algorithm, you will see that the prediction is not only for the next steep and that the prediction, is forward feed to another layer where it becomes the input of a more general and stable prediction. Also, there is strong evidence that prediction is the main (if not the only) function of the brain. I would recommend Hohwy's book (https://books.google.fr/books/about/The_Predictive_Mind.html?id=5QD2AQAAQBAJ&redir_esc=y https://books.google.fr/books/about/The_Predictive_Mind.html...) for an explanation of this idea.
versteegen 11y ago
Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the next word in a sentence based on a prediction of where this conversation is going) I think that predicting the next word in a sentence does actually get to the point of the issue, which is why the Hutter Prize [0] is defined the way it is. The human brain does: (1) operate sequentially on words (2) keeps an internal state that contains semantic information about what's been heard/seen (3) learns the semantic representation (4) is excellent at predicting the next word in a sentence, and continually does so (5) it's difficult to argue it's not learning to predict the next word in a sentence (ie optimised to that task). Looking at that list, I'd say the only differences between what the brain is doing and what algorithms like these (in general) are doing are just the particular algorithms for learning the semantic representations in working memory and performing prediction, and less importantly the form of the input (humans have more cues available, including taking actions to get more information) (Admittedly parsing a complicated sentence can require jumping backwards, but spoken language will have a very limited nesting level, and this just means including a small piece of the input in the working memory.) In the last week I've actually been reading about algorithms for this (though I didn't know NuPIC was applied to sentences, thanks very much for mentioning it). Aside from the range of RNN-based systems that mietek mentioned, here are a couple more recent papers, related to word2vec, on representing the semantic and syntactic content of a sentence as vectors: [1], [2]. Both are based on optimising for prediction of the next words in a sequence. Now I agree it's dubious to try to reduce a sentence to a small fixed length representation, whether that's e.g. 2400 real numbers in [2], or a pattern of neuron activations in a RNN, but from the experimental results (which admittedly aren't always so great) they seems to be encoding something meaningful, though far too much information gets thrown away. [0] http://prize.hutter1.net http://prize.hutter1.net [1] Quoc V. Le, Tomas Mikolov, 2014, Distributed Representations of Sentences and Documents, http://arxiv.org/abs/1405.4053 http://arxiv.org/abs/1405.4053 [2] Ryan Kiros et al, 2015, Skip-Thought Vectors, http://arxiv.org/abs/1506.06726 http://arxiv.org/abs/1506.06726 Edit: added a bit more about context
colhom 11y ago
A noamrl peosrn can uranesntdd this whuiott too much dftfliciuy . I scseput there is hveay rniaelce on the ecixtetapon of waht wdors lileky come nxet . I thnik taht liniestng to seceph wkors the same wya.

by mjpuser 11y ago

mietek 11y ago

> This works great for things with patterns, but it doesn't work with language. Sure it does. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ > Hawkins' mantra is sequence based memory. Note that Hawkins wrote “On intelligence” in 2005. Schmidhuber wrote “Learning complex extended sequences using the principle of history compression” in 1992, and “Long short-term memory”, with Hochreiter, in 1997. Today, a LSTM-based RNN can answer your Gmail. http://people.idsia.ch/~juergen/rnn.html http://people.idsia.ch/~juergen/rnn.html http://googleresearch.blogspot.com/2015/11/computer-respond-to-this-email.html http://googleresearch.blogspot.com/2015/11/computer-respond-...

mjpuser 11y ago
Sorry! To clarify that it doesn't work with language, I mean you can't ask it a question and then expect it to come up with an answer. For most meaningful questions, you can't sequence an answer from it's question in a general way. This might be in part because Hawkins isn't trying to mimic the brain, but learn about how it works to apply the algorithms it uses to our needs/wants.
- p1esk 11y ago
  you can't ask it a question and then expect it to come up with an answer Why not? If the model was trained on a large enough corpus of texts, it will have seen lots of answers to your question, or similar questions. It can, in principle, extract the important features of those answers, and present them to you as an answer. This is pretty much how the majority of humans would answer "the most meaningful" questions.
  - mjpuser 11y ago
    Nupic showcases it's "What did the fox eat" use case which uses cortical.io's SDRs of words. In this example it "asks the question" what does the fox eat after teach it what other animals eat, and without ever seeing fox before, it's able to accurately say what it eats since it groups foxes with coyotes, etc, which have a similar SDR. However, this is a sequence... They fed it sentences only in the form "a eats b" and then fed it "fox eats", and it replied "rodent" or whatever... The SDR's have to be sequences in order for the HTM to work. So if you ask "Guess how many balls are in this jug", you have to do an implied computation to guess how many possible balls fit inside the jug, and that implied computation is not the word sequence (or sdr sequence). Even if you gave it an infinite amount of similar questions and answers like this, it would never be able to figure out a general way to answer the question, which means you could always stump it by giving a different sized jug + different sized ball.
    - p1esk 11y ago
      H in HTM stands for "Hierarchical". This means, that it can, in principle, construct more general, more abstract SDRs (patterns) based on many low level SDRs it sees. Thus it's plausible to see forming of ideas out of sentences, or something similar. This is how it can build a world model, and then run your input through multiple levels of abstraction, as many as needed to give a high confidence answer.
      - mjpuser 11y ago
        My example was also spoon feeding questions and answers to the HTM. If you were to just give it a corpus of text, and then ask it a question about that text, it would not be able to formulate an answer that has any meaning, and follow proper English grammar. If you disagree, you can respond with a working example.

RobertoG 11y ago

If you study the Numenta algorithm, you will see that the prediction is not only for the next steep and that the prediction, is forward feed to another layer where it becomes the input of a more general and stable prediction. Also, there is strong evidence that prediction is the main (if not the only) function of the brain. I would recommend Hohwy's book (https://books.google.fr/books/about/The_Predictive_Mind.html?id=5QD2AQAAQBAJ&redir_esc=y https://books.google.fr/books/about/The_Predictive_Mind.html...) for an explanation of this idea.

versteegen 11y ago

Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the next word in a sentence based on a prediction of where this conversation is going) I think that predicting the next word in a sentence does actually get to the point of the issue, which is why the Hutter Prize [0] is defined the way it is. The human brain does: (1) operate sequentially on words (2) keeps an internal state that contains semantic information about what's been heard/seen (3) learns the semantic representation (4) is excellent at predicting the next word in a sentence, and continually does so (5) it's difficult to argue it's not learning to predict the next word in a sentence (ie optimised to that task). Looking at that list, I'd say the only differences between what the brain is doing and what algorithms like these (in general) are doing are just the particular algorithms for learning the semantic representations in working memory and performing prediction, and less importantly the form of the input (humans have more cues available, including taking actions to get more information) (Admittedly parsing a complicated sentence can require jumping backwards, but spoken language will have a very limited nesting level, and this just means including a small piece of the input in the working memory.) In the last week I've actually been reading about algorithms for this (though I didn't know NuPIC was applied to sentences, thanks very much for mentioning it). Aside from the range of RNN-based systems that mietek mentioned, here are a couple more recent papers, related to word2vec, on representing the semantic and syntactic content of a sentence as vectors: [1], [2]. Both are based on optimising for prediction of the next words in a sequence. Now I agree it's dubious to try to reduce a sentence to a small fixed length representation, whether that's e.g. 2400 real numbers in [2], or a pattern of neuron activations in a RNN, but from the experimental results (which admittedly aren't always so great) they seems to be encoding something meaningful, though far too much information gets thrown away. [0] http://prize.hutter1.net http://prize.hutter1.net [1] Quoc V. Le, Tomas Mikolov, 2014, Distributed Representations of Sentences and Documents, http://arxiv.org/abs/1405.4053 http://arxiv.org/abs/1405.4053 [2] Ryan Kiros et al, 2015, Skip-Thought Vectors, http://arxiv.org/abs/1506.06726 http://arxiv.org/abs/1506.06726 Edit: added a bit more about context

colhom 11y ago

A noamrl peosrn can uranesntdd this whuiott too much dftfliciuy . I scseput there is hveay rniaelce on the ecixtetapon of waht wdors lileky come nxet . I thnik taht liniestng to seceph wkors the same wya.