The 'big activity vector' language is partly a reference to the Vector Word Embedding stuff (see [1] for an explanation). The surprising thing about that is th

The 'big activity vector' language is partly a reference to the Vector Word Embedding stuff (see [1] for an explanation). The surprising thing about that is that it is possible to learn an embedded of individual words in a multi-dimensional vector space, such that (for example) vector(Queen)-vector(Woman) ~= vector(King)-vector(Man). Which is to say that there's a general 'royalty' direction within the vector space - and all this can be learned purely from seeing large amounts of English text (no 'traditional' supervised training). Perhaps a 'god' could identify the meaning of each direction in the space (or region of words), but the big Machine Learning labs (Google, Baidu, Montreal, Stanford, Facebook, etc) are proving that purely manipulating the vectors in the abstract works really well. In addition to the 'word vectors' as inputs, the RNNs illustrated are also iterating over an internal state (flowing from left to right though the same network for each new word) - and this internal state is also an embedding of some kind. But it's going to be very difficult to decipher what each dimension here represents, as it's being built purely as a function of the input word vectors, its own previous state and a NN with initially random weights. Now, although actual 'brain experiments' have shown that individual neuron (or local clusters) apparently light up when particular thoughts are had (alternatively, cause thoughts to be had), each cluster seems likely to be just one aspect of (say) 'dogginess'. So, one area will correspond to the smell of dogs, others to wet noses, others to being outdoors (i.e. all aspects of the overall 'dogginess' concept) - but these things will all overlap in multiple ways with other concept 'vectors'. Which is how huge spaces of ideas are searched in parallel, rather than sequentially (using, say, an is_doggy_quality symbol). There are also parallels here with the Numenta Sparse Distributed Representations [2]. Overall, this presentation seems to be probing at the frontier of what works, and how to leverage that up into something that's more about 'general thinking' rather than pattern matching. It also appears to be a thought-piece, rather than a conference presentation (though, of course, Hinton deserves to be heard on just about anything in NNs, IMHO). [1] http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ http://colah.github.io/posts/2014-07-NLP-RNNs-Representation... [2] https://github.com/numenta/nupic/wiki/Sparse-Distributed-Representations https://github.com/numenta/nupic/wiki/Sparse-Distributed-Rep...