20 ms·
The Thousand Brains Theory of Intelligence (2019)
- java-man 6y agoTitle needs (2019). A new book by Jeff Hawkins is coming out [0]. [0] https://www.amazon.com/gp/product/1541675819 https://www.amazon.com/gp/product/1541675819
- VladimirGolovin 6y agoInteresting. One of the blurbs got my attention: "Brilliant....It works the brain in a way that is nothing short of exhilarating." ― Richard Dawkins.
- cjauvin 6y agoHis previous book (On Intelligence) is very stimulating and contains many ideas that have interesting connections with Deep Learning. I'm looking forward to read this new one.
- azernik 6y agoYou know it's an exciting field when something 1.5-2 years old needs a date on it.
- abeppu 6y agoI kept expecting terms like "ensemble" and "feature bagging" to pop up, but they didn't. How are others mentally mapping these concept back to ML?
- gojomo 6y agoAlso "dropout".
- sdenton4 6y agoYeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Each training step updates a different random subnetwork. (...Keeping in mind that two random subnetworks can share lots of weights...) Then at inference time, you use the full network, which now acts as an ensemble of all of the subnetworks. You could view it as 'thousand brains' with lots of weight sharing for efficiency+regularization.
- ur-whale 6y agoMuch like backprop, I'm not sure dropout can be mapped back to the way an actual brain works.
- dcx 6y agoWait, why can't you? Backprop is basically adjusting weights to get the model closer to making correct predictions. Isn't that basically how the brain does it? (Though I'm sure the way the brain adjusts its weights uses a different approximation of differentiation)
- abeppu 6y agoI think the main issues are: - there's not a "correct" training prediction - there's not a "final layer"
- ur-whale 6y agoBackprop is not simply about adjusting the weights (as a matter of fact, you can argue that any training method is about adjusting the weights). It's about computing the amount by which you adjust the weight. And unless there's been a major development in neuroscience that I'm not aware of, backprop is not the way the brain does it.
- gojomo 6y ago"Thousand Brains" sounds a lot like "Society of Mind": https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind (Perhaps, with a bit more "arbitrary horizontal diversification based on subsets of inputs" rather than mainly "specialization by function".)
- bglusman 6y agoYeah, came here to say this also, my memory of the book is fuzzy but I have a copy somewhere and is where my mind went right away.
- Medox 6y agoReminds me of the last part from The Secret Life of Chaos [1] (after 51:00) and their 100 random brains. [1] https://www.dailymotion.com/video/xv1j0n https://www.dailymotion.com/video/xv1j0n
- gamegoblin 6y agoHighly recommend the Lex Fridman interview to listen to Jeff explain live: https://www.youtube.com/watch?v=-EVqrDlAqYo https://www.youtube.com/watch?v=-EVqrDlAqYo While Numenta doesn't have the amazing results of DeepMind and friends, I think they are doing really great stuff in the space of biologically plausible intelligence.
- jcims 6y agoSeconded. Even if you aren’t ‘into’ this stuff, Jeff has some interesting ideas on how the brain works that map pretty well to my personal subjective experience. (Not saying that makes them true.)
- ur-whale 6y agoI struggle to understand why this new theory is in any way original. The way an artificial neural network actually functions can most certainly be interpreted as multiple sub-systems "voting" to reach a conclusion. You can even explicitly design architectures to perform that task exclusively (mixture of experts). Since ANNs were loosely designed on the way we assume the brain works ... back in the 50's ... how is this an original idea? Am I missing something?
- wwarner 6y agoThey're proposing a theory of how the brain's cortex must work. Check out some interviews with Jeff Hawkins of Numenta, such as the forementioned interview with Lex Fridman. One essential insight of theirs is that the cortex is physiologically the same everywhere, so any differences between regions would be caused by what happens to be connected to them, rather than something that is inherent to them. This idea of 1000 brains is sort of along those lines.
- lostmsu 6y agoTo me this still does not explain what is original or interesting about that idea. Artificial neural networks are "physiologically" the same everywhere too, and have been like that from the inception.
- galaxyLogic 6y agoIf you train a neural network you get some results. But presumably if you train a 1000 separate neural networks on different measurements made of the same target, and then let those thousand models vote, isn't that a new kind of learning mechanism, compared to having a single neural network that always takes in all the inputs? Further each of the thousand neural nets could have a different learning algorithm.
- lostmsu 6y ago> let those thousand models vote Soon after you start thinking how voting can be implemented, you will quickly realize, that one of the best ways to do it is to connect each of 1000 networks to each of N outputs with some sort of weights (in trivial version - uniform), which effectively turns 1000 networks into 1 network with extra N unit layer, and no weight reuse (e.g. probably a disadvantage) in all but the last layer.
- cmarschner 6y agoI wish they would be working less like a company and would engage more with the research community on the topic. The topic of On Intelligence, Hierarchical Temporal Memory (HTM) was never open sourced, also didn’t have published math, and was thus treated by the community as weird and not actionable, and in the end ignored.
- p1esk 6y agoIt is open source, and they have at least a dozen papers published.
- greyface- 6y agoThey also post recordings of many of their internal research meetings on YouTube: https://www.youtube.com/c/NumentaTheory/videos https://www.youtube.com/c/NumentaTheory/videos
- cmarschner 6y agoOk then my information is outdated. It’s been a while.
- dublin 6y agoIt is quite likely that the brain's relationships are not just 2- or 3-dimensional, but rather very highly multidimensionally interconnected. When neural nets seriously hit the scene 30-odd years ago (I read a lot of the early papers, even having to mail off to get some in those pre-Internet days), no one seriously thought they were the way the brain works, just that they presented the opportunity to simulate a tiny slice of the way brains might work. As Ted Nelson so perfectly puts it, "Everything is deeply intertwingled."
- dang 6y agoIf curious, past threads: Numenta Platform for Intelligent Computing - https://news.ycombinator.com/item?id=24613866 https://news.ycombinator.com/item?id=24613866 - Sept 2020 (23 comments) Jeff Hawkins: Thousand Brains Theory of Intelligence [video] - https://news.ycombinator.com/item?id=20326396 https://news.ycombinator.com/item?id=20326396 - July 2019 (94 comments) The Thousand Brains Theory of Intelligence - https://news.ycombinator.com/item?id=19311279 https://news.ycombinator.com/item?id=19311279 - March 2019 (37 comments) Jeff Hawkins Is Finally Ready to Explain His Brain Research - https://news.ycombinator.com/item?id=18214707 https://news.ycombinator.com/item?id=18214707 - Oct 2018 (69 comments) IBM creates a research group to test Numenta, a brain-like AI software - https://news.ycombinator.com/item?id=9401697 https://news.ycombinator.com/item?id=9401697 - April 2015 (19 comments) Jeff Hawkins: Brains, Data, and Machine Intelligence [video] - https://news.ycombinator.com/item?id=8804824 https://news.ycombinator.com/item?id=8804824 - Dec 2014 (15 comments) Jeff Hawkins on the Limitations of Artificial Neural Networks - https://news.ycombinator.com/item?id=8544561 https://news.ycombinator.com/item?id=8544561 - Nov 2014 (16 comments) Numenta Platform for Intelligent Computing - https://news.ycombinator.com/item?id=8062175 https://news.ycombinator.com/item?id=8062175 - July 2014 (25 comments) Numenta open-sourced their Cortical Learning Algorithm - https://news.ycombinator.com/item?id=6304363 https://news.ycombinator.com/item?id=6304363 - Aug 2013 (20 comments) Palm founder Jeff Hawkins on neurology, big data, and the future of AI - https://news.ycombinator.com/item?id=5917481 https://news.ycombinator.com/item?id=5917481 - June 2013 (6 comments) Numenta releases brain-derived learning algorithm package NuPIC - https://news.ycombinator.com/item?id=5814382 https://news.ycombinator.com/item?id=5814382 - June 2013 (59 comments) The Grok prediction engine from Numenta announced - https://news.ycombinator.com/item?id=3933631 https://news.ycombinator.com/item?id=3933631 - May 2012 (25 comments) Jeff Hawkins talk on modeling neocortex and its impact on machine intelligence - https://news.ycombinator.com/item?id=1945428 https://news.ycombinator.com/item?id=1945428 - Nov 2010 (27 comments) Jeff Hawkins' "On Intelligence" and Numenta startup - https://news.ycombinator.com/item?id=59012 https://news.ycombinator.com/item?id=59012 - Sept 2007 (3 comments) The Thinking Machine: Jeff Hawkins's new startup, Numenta - https://news.ycombinator.com/item?id=3539 https://news.ycombinator.com/item?id=3539 - March 2007 (3 comments)
- jonplackett 6y ago> A three-dimensional representation of objects also provides the basis for learning compositional structure, i.e. how objects are composed of other objects arranged in particular ways I found this particularly interesting. I have a young daughter and I find it fascinating how quickly and reliably she can learn an object and then recognise others in a way a computer just can’t do at all. Eg see very rough line drawing of an object, then recognise that animal in real life. It’s also interesting the way I can explain new objects or animals based on one she knows and then have her recognise them easily. Eg a tiger is like a lion but has no mane and does have stripes. This comes very naturally when teaching and learning and feels very uniquely ‘human’ compared to the rigidity of computer vision I’ve seen so far.
- ckosidows 6y agoHave you seen Open AI's DALL•E? https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ Kind of does something similar to what you describe, right? This project blows my mind, by the way. I still think it's the coolest thing to come out of AI research so far.
- chrisweekly 6y agoWOW!! I'm astounded. Zero-shot visual reasoning?? As an emergent capability?? Um. wat. >"GPT-3 can be instructed to perform many kinds of tasks solely from a description and a cue to generate the answer supplied in its prompt, without any additional training. For example, when prompted with the phrase “here is the sentence ‘a person walking his dog in the park’ translated into French:”, GPT-3 answers “un homme qui promène son chien dans le parc.” This capability is called zero-shot reasoning. We find that DALL·E extends this capability to the visual domain, and is able to perform several kinds of image-to-image translation tasks when prompted in the right way."
- tablespoon 6y ago> Have you seen Open AI's DALL•E? https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ Is there any way to try it out easily? Everything on that page looked like it was pre-rendered (constrained-choice mad libs).
- meroes 6y agoWhy have so many hypotheses failed when it comes to the mind? We understand quantum scale effects and measure and predict them to 17 decimals places of precision. How can we not know what is going on in the brain? We know what is going on in supercolliders when trillions of particles scatter (we only capture a tiny fraction of the data but still we know all the mechanisms available to the standard model). Surely we can track any and every physical mechanism in the brain "accessible" to consciousness to build itself up from assuming it is physical. How can we still not take that set of physical mechanisms and build a lasting model of consciousness. I have a personal bet that no progress will be made on consciousness in my lifetime. Maybe one of the theories we have is correct and we just can't test it correctly. I could easily be wrong. But it seems like we lack the imagination, not the technical ability, to find the answer. And not much has changed from my outside perspective in 30 years. Look, I'm genuinely puzzled by science's inability to have a working model of consciousness. I'm not asking rhetorically why we have no lasting models, I don't know what the reason could be. To me these are even more "woo" than string theory which gets lambasted around the popular alt big-science channels.
- aeternum 6y agoWe can't even model most two-element chemical systems using QM as the state space just becomes too complex. Having a model does not mean you can necessarily run the computations to sufficient accuracy to make good predictions. Even with something as well-understood as gravity, N-body problems are still quite difficult and calculations must be numerically approximated. This method works fine for most systems but diverges rapidly if a chaotic perturbation is present.
- andyxor 6y agoJeff Hawkins has a great series of open workshops on computational cognitive models (with Marcus Lewis and the rest of Numenta team), check out Numenta on Twitter. This open seminar format is kind of unique in the industry. Recently they talked about grid cells models: https://twitter.com/Numenta/status/1357836938955218945 https://twitter.com/Numenta/status/1357836938955218945 and reviewing the new 'Tolman-Eichenbaum Machine' memory model by James Whittington' lab: https://twitter.com/Numenta/status/1362187375900651526 https://twitter.com/Numenta/status/1362187375900651526
- flylikeabanana 6y agoI wonder if this works for efferent behavior too. Perhaps when throwing a rock, we actually throw it a thousand times in our brain before letting go of it, as feedback from our muscles converges with our perception of ballistic forces. Like running a thousand simulations and picking the best one, as the event is happening.
- ilaksh 6y agoIn case anyone reads this, also see Dileep George and Andy Clark.