9 ms·

Why Does the Neocortex Have Columns? A Theory of Learning Structure of the World [pdf]

237 points by blacksmythe 9y ago

skybrian 9y ago
There is also a summary at: https://blog.acolyer.org https://blog.acolyer.org
- mikhailfranco 9y ago
  ... probably not by pure coincidence. p.s. specific dated link https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-have-columns-a-theory-of-learning-the-structure-of-the-world/ https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-h...
brad0 9y ago
Great to see numenta in the news again. On Intelligence was the book that got me excited about biologically based machine learning. The methods that in that book is very different to anything else I’ve seen in current “trendy” ML.
benthehorn12 9y ago
falksjf
deleted 9y ago
[deleted]
electrograv 9y ago
Full paper here: https://www.biorxiv.org/content/biorxiv/early/2017/09/28/162263.full.pdf https://www.biorxiv.org/content/biorxiv/early/2017/09/28/162... I always enjoy reading analysis/ideas/intuitions about how the brain works, because it provides inspiration for machine learning improvements that can be applied in the real world. That said, I’m still optimistically waiting for Numenta (and Geoff Hinton’s capsule theory) to set the new bar at one of the many difficult image/speech/language/etc recognition challenges. Ideas are great, but at the end of the day science moves forward when we measure our ideas against reality. (To the credit of this paper, it does make a series of predictions, though those seem extremely difficult to measure in biological systems for the time being.)
adamnemecek 9y ago
I'm probably talking out of my ass but I'm somewhat suspect of a cortex having straight up columns. I'm curious whether these are artifacts of the fact that linear algebra seems to be the dominant algebra in ML/modeling of human perception. Recently I've been dipping my feet into geometric algebra which seems to be the superior algebra for just about anything you can think of (human perception but also like all of physics, Maxwell's 4 equations are reduced to a single equation in GA) and it's particularly better for reasoning about spaces which this seems to be all about. And unlike linear algebra it actually makes sense (e.g. why is cross product only in three dimensions?, wtf are determinants esp. in the context of matrix division all about?). This blog post introduces GA and talks about it's relationship to human perception. https://slehar.wordpress.com/2014/03/18/clifford-algebra-a-visual-introduction/ https://slehar.wordpress.com/2014/03/18/clifford-algebra-a-v...
- gowld 9y ago
  Linear algebra is equivalent to geometric algebra so you wouldn't find columns less likely in GA. Also, GA is usually hyped by non-mathematicians -- people who don't understand that it's just an alternative notation that doesn't add much to linear algebra. > Clifford Algebra, a.k.a. Geometric Algebra, is a most extraordinary synergistic confluence of a diverse range of specialized mathematical fields > It is the very simplicity and generality of Clifford Algebra that confirms its “truth” kook alarms ringing
  - adamnemecek 9y ago
    > Linear algebra is equivalent to geometric algebra That depends on your definition of equivalent. They have the same computational power but things make sense in one and less in the other. > Also, GA is usually hyped by non-mathematician What a loaded statement. > kook alarms ringing You aren't the first one to say it. Nonetheless, the author is a researcher at Harvard it seems http://cns-alumni.bu.edu/~slehar/Lehar.html http://cns-alumni.bu.edu/~slehar/Lehar.html Don't let the form fool you. You can also read any of the many books on the topic.
    - 9y ago
rhyolight_ 9y ago
Here are some video resources to help explain this theory: - https://www.youtube.com/watch?v=BvJJn9VS4rk https://www.youtube.com/watch?v=BvJJn9VS4rk - https://www.youtube.com/watch?v=-h-cz7yY-G8 https://www.youtube.com/watch?v=-h-cz7yY-G8
- pault 9y ago
  Here's another very interesting talk from Jeff Hawkins about modelling the neocortex: https://www.youtube.com/watch?v=izO2_mCvFaw https://www.youtube.com/watch?v=izO2_mCvFaw
Upvoter33 9y ago
The world needs more work being done in the way Hawkins and co. are doing it, and less in the mold of most deep learning/ML work. Why? He's actually trying to connect building intelligent machines with biology. This is a huge problem, but so few are working on it. Rather, we are all distracted by deep learning because of its recent successes in very specific problem areas. In a few years, when we run into its limits, Hawkins and people doing work like this will have a chance to shine (if they produce something that works, of course).
- pizza 9y ago
  But wait a minute... the world is already doing quite a good deal of work on this type of problem. Maybe not enough, but it's been goin on https://en.wikipedia.org/wiki/Computational_neuroscience https://en.wikipedia.org/wiki/Computational_neuroscience
- bbctol 9y ago
  Deepmind does a ton of work in meat neuroscience, and are actually publishing useful research in the field (a recent if somewhat controversial review at http://www.cell.com/neuron/fulltext/S0896-6273(17)30509-3 http://www.cell.com/neuron/fulltext/S0896-6273(17)30509-3) And they're about the most prominent deep learning group I can think of.
- dontreact 9y ago
  Computer vision and NLP are pretty broad problem areas with many applications. These technologies are already powerful enough to influence the world in many positive ways. I wouldn’t say that Actually realizing this potential is a “distraction”
platz 9y ago
Skeptical. Previously the brain was a machine consisting of drives and pulleys, just like the state of technology of the day. Now the brain is a computer that runs deep learning models. It would be one thing to just say the cortex has columns. It's another to go on and model what those columns are doing with linear algebra. The coincidence that this picture emerges at the same time as current fads in tech is too great to ignore I am more interested in the work that is identifying different kinds of cells e.g place cells
- thwd 9y ago
  > place cells Suiting, given that your username means "place" in German :)
- platz 9y ago
  Alternatively if your real goal is just to develop some novel ML techniques, it would be better to not make claims about how the brain works and instead just try to draw inspiration using these thought experiments more as a 'muse'
- freeflight 9y ago
  > The coincidence that this picture emerges at the same time as current fads in tech is too great to ignore It's no coincidence, I see it as a kind of reframing the issue from a different perspective/approach and for that, we use whatever seems the best framework of explanation we have at hand during that time. Gotta start somewhere, can't try to paint a picture without a frame and at least some colors. One can easily see the evolution how the frameworks we use keep getting more complex, from drives and pulleys to computers and NNs, so there clearly is some progress in how we are trying to describe the workings of the brain/consciousness. In the end, it's also about what's the actual goal here: Trying to understand the human brain/consciousness or trying to artificially create "consciousness" or rather something resembling it. The later doesn't necessarily require the former.
  - rhyolight_ 9y ago
    Not consciousness, but intelligence. Big difference.
Cacti 9y ago
Jeff Hawkins gets shit on a lot in ML because his theories haven’t produced models with results as good as the mainstream approach, but I’m glad they keep working at it and keep coming up with interesting ideas. Too much of ML these days is about some NN model that does 0.n% better than SOTA on some specific task. Then you change one tiny parameter and he entire thing breaks, and it turns out we didn’t understand why it was working at all.
- daveguy 9y ago
  I do think his reasoning and focus of what's wrong with ML today is good. Online learning, sequence learning and "motor" feedback are all critical. Unfortunately he came up with this HTM and appears to be trying to cram all advances into the same framework rather than stepping back and thinking about how those critical features could be implemented differently. Also as far as I know hebbian learning (like HTM) has achieved ~90% accuracy on MNIST while SOA is above 99% accuracy. So it's not a small gap between HTM and state of the art. If HTM was near 98% it would be impressive (learning is faster and transfer better). I think alternatives or hybrids are necessary, but he seems stuck on his pet project. That said, the principles and "what's important for intelligence" I think he is right on target and articulates those aspects well.
- chplushsieh 9y ago
  Jeff Hawkins gets shit on a lot in ML because his theories haven’t produced models with results as good as the mainstream approach, but I’m glad they keep working at it and keep coming up with interesting ideas. I just ran across a paper [1] from Numenta on Time Series Anomaly Detection using HTM last night which provide a benchmark[2] with some existing approachs. (But it seems to me there is no NN based approach in them. ) [1] Unsupervised real-time anomaly detection for streaming data http://www.sciencedirect.com/science/article/pii/S0925231217309864 http://www.sciencedirect.com/science/article/pii/S0925231217... [2] https://github.com/numenta/NAB https://github.com/numenta/NAB
- davesque 9y ago
  From a mostly outside point of view, I've wondered a lot about this. It seems a lot of ML/DL research that goes on nowadays amounts to educated guessing and tinkering i.e. "Based on our notion of what seemed right, we tried this other configuration of layers and/or activation functions and look; it worked". I imagine at some point that the kind of research discussed in the linked post will begin to pan out more and more and will eventually lead to another period of rapid progress like that which followed Hinton's work on DL.
nopacience 9y ago
Understand how the brain works means understand how we the humans see and understand the world and ourselfs within it. The brain organically and selectivelly selects what to learn and which information shou ld be retained. Our brain receives training since birthdays. Then the family implements some of their own training then school and world. The brain never stops. Great job Numenta !!
Double_Cast 9y ago
> Error! Problem, or Page Not Found > Sorry, the page you were looking for does not exist. Link is broken. From browsing, I believe the correct link is https://numenta.com/papers-videos-and-more/resources/layers-and-columns/ https://numenta.com/papers-videos-and-more/resources/layers-...
- Avery3R 9y ago
  The link works fine if you disable javascript. Still weird though
mholt 9y ago
I'm getting "Error! Problem, or Page Not Found Sorry, the page you were looking for does not exist." Edit: Weird, I just closed my browser and tried again, and the article looks like it flashed into the screen then was replaced by the error message. Happens repeatedly on Chrome on Android...
- dang 9y ago
  I'm seeing that too. Ok, we'll change the URL from https://numenta.com/papers/why-does-the-neocortex-have-layers-and-columns/ https://numenta.com/papers/why-does-the-neocortex-have-layer... to the pdf of the paper. There's also a summary at https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-have-columns-a-theory-of-learning-the-structure-of-the-world/ https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-h... that other commenters have mentioned.
deleted 9y ago
[deleted]

237 points by blacksmythe 9y ago

skybrian 9y ago

There is also a summary at: https://blog.acolyer.org https://blog.acolyer.org

mikhailfranco 9y ago
... probably not by pure coincidence. p.s. specific dated link https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-have-columns-a-theory-of-learning-the-structure-of-the-world/ https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-h...

brad0 9y ago

Great to see numenta in the news again. On Intelligence was the book that got me excited about biologically based machine learning. The methods that in that book is very different to anything else I’ve seen in current “trendy” ML.

benthehorn12 9y ago

falksjf

deleted 9y ago

[deleted]

electrograv 9y ago

Full paper here: https://www.biorxiv.org/content/biorxiv/early/2017/09/28/162263.full.pdf https://www.biorxiv.org/content/biorxiv/early/2017/09/28/162... I always enjoy reading analysis/ideas/intuitions about how the brain works, because it provides inspiration for machine learning improvements that can be applied in the real world. That said, I’m still optimistically waiting for Numenta (and Geoff Hinton’s capsule theory) to set the new bar at one of the many difficult image/speech/language/etc recognition challenges. Ideas are great, but at the end of the day science moves forward when we measure our ideas against reality. (To the credit of this paper, it does make a series of predictions, though those seem extremely difficult to measure in biological systems for the time being.)

adamnemecek 9y ago

I'm probably talking out of my ass but I'm somewhat suspect of a cortex having straight up columns. I'm curious whether these are artifacts of the fact that linear algebra seems to be the dominant algebra in ML/modeling of human perception. Recently I've been dipping my feet into geometric algebra which seems to be the superior algebra for just about anything you can think of (human perception but also like all of physics, Maxwell's 4 equations are reduced to a single equation in GA) and it's particularly better for reasoning about spaces which this seems to be all about. And unlike linear algebra it actually makes sense (e.g. why is cross product only in three dimensions?, wtf are determinants esp. in the context of matrix division all about?). This blog post introduces GA and talks about it's relationship to human perception. https://slehar.wordpress.com/2014/03/18/clifford-algebra-a-visual-introduction/ https://slehar.wordpress.com/2014/03/18/clifford-algebra-a-v...

gowld 9y ago
Linear algebra is equivalent to geometric algebra so you wouldn't find columns less likely in GA. Also, GA is usually hyped by non-mathematicians -- people who don't understand that it's just an alternative notation that doesn't add much to linear algebra. > Clifford Algebra, a.k.a. Geometric Algebra, is a most extraordinary synergistic confluence of a diverse range of specialized mathematical fields > It is the very simplicity and generality of Clifford Algebra that confirms its “truth” kook alarms ringing
- adamnemecek 9y ago
  > Linear algebra is equivalent to geometric algebra That depends on your definition of equivalent. They have the same computational power but things make sense in one and less in the other. > Also, GA is usually hyped by non-mathematician What a loaded statement. > kook alarms ringing You aren't the first one to say it. Nonetheless, the author is a researcher at Harvard it seems http://cns-alumni.bu.edu/~slehar/Lehar.html http://cns-alumni.bu.edu/~slehar/Lehar.html Don't let the form fool you. You can also read any of the many books on the topic.
  - 9y ago

rhyolight_ 9y ago

Here are some video resources to help explain this theory: - https://www.youtube.com/watch?v=BvJJn9VS4rk https://www.youtube.com/watch?v=BvJJn9VS4rk - https://www.youtube.com/watch?v=-h-cz7yY-G8 https://www.youtube.com/watch?v=-h-cz7yY-G8

pault 9y ago
Here's another very interesting talk from Jeff Hawkins about modelling the neocortex: https://www.youtube.com/watch?v=izO2_mCvFaw https://www.youtube.com/watch?v=izO2_mCvFaw

Upvoter33 9y ago

The world needs more work being done in the way Hawkins and co. are doing it, and less in the mold of most deep learning/ML work. Why? He's actually trying to connect building intelligent machines with biology. This is a huge problem, but so few are working on it. Rather, we are all distracted by deep learning because of its recent successes in very specific problem areas. In a few years, when we run into its limits, Hawkins and people doing work like this will have a chance to shine (if they produce something that works, of course).

pizza 9y ago
But wait a minute... the world is already doing quite a good deal of work on this type of problem. Maybe not enough, but it's been goin on https://en.wikipedia.org/wiki/Computational_neuroscience https://en.wikipedia.org/wiki/Computational_neuroscience
bbctol 9y ago
Deepmind does a ton of work in meat neuroscience, and are actually publishing useful research in the field (a recent if somewhat controversial review at http://www.cell.com/neuron/fulltext/S0896-6273(17)30509-3 http://www.cell.com/neuron/fulltext/S0896-6273(17)30509-3) And they're about the most prominent deep learning group I can think of.
dontreact 9y ago
Computer vision and NLP are pretty broad problem areas with many applications. These technologies are already powerful enough to influence the world in many positive ways. I wouldn’t say that Actually realizing this potential is a “distraction”

platz 9y ago

Skeptical. Previously the brain was a machine consisting of drives and pulleys, just like the state of technology of the day. Now the brain is a computer that runs deep learning models. It would be one thing to just say the cortex has columns. It's another to go on and model what those columns are doing with linear algebra. The coincidence that this picture emerges at the same time as current fads in tech is too great to ignore I am more interested in the work that is identifying different kinds of cells e.g place cells

thwd 9y ago
> place cells Suiting, given that your username means "place" in German :)
platz 9y ago
Alternatively if your real goal is just to develop some novel ML techniques, it would be better to not make claims about how the brain works and instead just try to draw inspiration using these thought experiments more as a 'muse'
freeflight 9y ago
> The coincidence that this picture emerges at the same time as current fads in tech is too great to ignore It's no coincidence, I see it as a kind of reframing the issue from a different perspective/approach and for that, we use whatever seems the best framework of explanation we have at hand during that time. Gotta start somewhere, can't try to paint a picture without a frame and at least some colors. One can easily see the evolution how the frameworks we use keep getting more complex, from drives and pulleys to computers and NNs, so there clearly is some progress in how we are trying to describe the workings of the brain/consciousness. In the end, it's also about what's the actual goal here: Trying to understand the human brain/consciousness or trying to artificially create "consciousness" or rather something resembling it. The later doesn't necessarily require the former.
- rhyolight_ 9y ago
  Not consciousness, but intelligence. Big difference.

Cacti 9y ago

Jeff Hawkins gets shit on a lot in ML because his theories haven’t produced models with results as good as the mainstream approach, but I’m glad they keep working at it and keep coming up with interesting ideas. Too much of ML these days is about some NN model that does 0.n% better than SOTA on some specific task. Then you change one tiny parameter and he entire thing breaks, and it turns out we didn’t understand why it was working at all.

daveguy 9y ago
I do think his reasoning and focus of what's wrong with ML today is good. Online learning, sequence learning and "motor" feedback are all critical. Unfortunately he came up with this HTM and appears to be trying to cram all advances into the same framework rather than stepping back and thinking about how those critical features could be implemented differently. Also as far as I know hebbian learning (like HTM) has achieved ~90% accuracy on MNIST while SOA is above 99% accuracy. So it's not a small gap between HTM and state of the art. If HTM was near 98% it would be impressive (learning is faster and transfer better). I think alternatives or hybrids are necessary, but he seems stuck on his pet project. That said, the principles and "what's important for intelligence" I think he is right on target and articulates those aspects well.
chplushsieh 9y ago
Jeff Hawkins gets shit on a lot in ML because his theories haven’t produced models with results as good as the mainstream approach, but I’m glad they keep working at it and keep coming up with interesting ideas. I just ran across a paper [1] from Numenta on Time Series Anomaly Detection using HTM last night which provide a benchmark[2] with some existing approachs. (But it seems to me there is no NN based approach in them. ) [1] Unsupervised real-time anomaly detection for streaming data http://www.sciencedirect.com/science/article/pii/S0925231217309864 http://www.sciencedirect.com/science/article/pii/S0925231217... [2] https://github.com/numenta/NAB https://github.com/numenta/NAB
davesque 9y ago
From a mostly outside point of view, I've wondered a lot about this. It seems a lot of ML/DL research that goes on nowadays amounts to educated guessing and tinkering i.e. "Based on our notion of what seemed right, we tried this other configuration of layers and/or activation functions and look; it worked". I imagine at some point that the kind of research discussed in the linked post will begin to pan out more and more and will eventually lead to another period of rapid progress like that which followed Hinton's work on DL.

nopacience 9y ago

Understand how the brain works means understand how we the humans see and understand the world and ourselfs within it. The brain organically and selectivelly selects what to learn and which information shou ld be retained. Our brain receives training since birthdays. Then the family implements some of their own training then school and world. The brain never stops. Great job Numenta !!

Double_Cast 9y ago

> Error! Problem, or Page Not Found > Sorry, the page you were looking for does not exist. Link is broken. From browsing, I believe the correct link is https://numenta.com/papers-videos-and-more/resources/layers-and-columns/ https://numenta.com/papers-videos-and-more/resources/layers-...

Avery3R 9y ago
The link works fine if you disable javascript. Still weird though

mholt 9y ago

I'm getting "Error! Problem, or Page Not Found Sorry, the page you were looking for does not exist." Edit: Weird, I just closed my browser and tried again, and the article looks like it flashed into the screen then was replaced by the error message. Happens repeatedly on Chrome on Android...

dang 9y ago
I'm seeing that too. Ok, we'll change the URL from https://numenta.com/papers/why-does-the-neocortex-have-layers-and-columns/ https://numenta.com/papers/why-does-the-neocortex-have-layer... to the pdf of the paper. There's also a summary at https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-have-columns-a-theory-of-learning-the-structure-of-the-world/ https://blog.acolyer.org/2017/10/25/why-does-the-neocortex-h... that other commenters have mentioned.

deleted 9y ago

[deleted]