3 ms·

Deep learning is mostly irrelevant for AGI but the best part of the article is bringing up the "recursive process called Merge”. This Merge [0] is called "chun

Deep learning is mostly irrelevant for AGI but the best part of the article is bringing up the "recursive process called Merge”. This Merge [0] is called "chunking" in cognitive psychology [1, 2], first mentioned in classic paper "The Magical Number Seven" by George A. Miller [3]. In the original Chomsky work[0] it is buried so deep in linguistics jargon it's easy to miss the centrality of this concept, which is the essence of generalization capability in biological mind. It's the JOIN in Leslie Valiant LINK/JOIN model [4, 5]: "The first basic function, JOIN, implements memory formation of a new item in terms of two established items: If two items A and B are already represented in the neural system, the task of JOIN is to modify the circuit so that at subsequent times there is the representation of a new item C that will fire if and only if the representations of both A and B are firing." Papadimitriou & Vempala [6] extend it to "predictive join" (PJOIN) model. Edit: As I think about it deep learning might be useful in implementing this "Merge" by doing nonlinear PCA (Principal Component Analysis) via stacked sparse autoencoders, kind of like in that "Cat face detection" paper by Quoc Le [7]. The only thing missing is hierarchical memory representation for those principal components, where NEW objects emerge by joining most similar existing objects. [0] https://en.wikipedia.org/wiki/Merge_(linguistics) https://en.wikipedia.org/wiki/Merge_(linguistics) [1] https://en.wikipedia.org/wiki/Chunking_(psychology) https://en.wikipedia.org/wiki/Chunking_(psychology) [2] http://www.columbia.edu/~nvg1/Wickelgren/papers/1979cWAW.pdf http://www.columbia.edu/~nvg1/Wickelgren/papers/1979cWAW.pdf [3] https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus... [4] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.208.8491&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.208... [5] https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/0195126688 https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/019... [6] https://arxiv.org/pdf/1412.7955.pdf https://arxiv.org/pdf/1412.7955.pdf [7] https://ieeexplore.ieee.org/abstract/document/6639343 https://ieeexplore.ieee.org/abstract/document/6639343

mcswell 6y ago
"the task of JOIN is to modify the circuit so that at subsequent times there is the representation of a new item C that will fire if and only if the representations of both A and B are firing." What that describes sounds like what in linguistic (syntactic) terms one would call conjunction. I don't recall whether Chomsky's recent theories (which come under the rubrik of "minimalism") count conjunction as a type of Merge, but what he's more concerned with is subordination, which is quite a different thing from conjunction, given that subordination--but not conjunction--results in hierarchical structures. And for the record, I don't think Merge in the linguistic sense has anything to do with "chunking", much less the "magical number seven." But then I don't know much about chunking.

mcswell 6y ago

"the task of JOIN is to modify the circuit so that at subsequent times there is the representation of a new item C that will fire if and only if the representations of both A and B are firing." What that describes sounds like what in linguistic (syntactic) terms one would call conjunction. I don't recall whether Chomsky's recent theories (which come under the rubrik of "minimalism") count conjunction as a type of Merge, but what he's more concerned with is subordination, which is quite a different thing from conjunction, given that subordination--but not conjunction--results in hierarchical structures. And for the record, I don't think Merge in the linguistic sense has anything to do with "chunking", much less the "magical number seven." But then I don't know much about chunking.