4 ms·

Recently Matt Taylor from numenta had a reddit AMA (https://www.reddit.com/r/artificial/comments/6beeqj/5182017_1200_pm_pst_iama_with_matt_taylor_numenta/ https

Recently Matt Taylor from numenta had a reddit AMA (https://www.reddit.com/r/artificial/comments/6beeqj/5182017_1200_pm_pst_iama_with_matt_taylor_numenta/ https://www.reddit.com/r/artificial/comments/6beeqj/5182017_...). I was disappointed to find out that numenta does not really have collaborations with neuroscientists to test or adapt their theories. Their theories in general are not well known to computational neuroscientists either. In that sense, i m not even sure about the authority of numenta on neocortical theories. For example, the article mentions the ability of clustered synapses to act independently, but , on the one hand, it has been shown independent dendrites can be approximated as an extra neural network layer (so they ARE covered by today's ANN approximation) , and OTOH there s a number of papers showing that synaptic clustering does not exist in sensory areas. And learning by rewiring is basically the introduction of random connections which persist only if their weight increases enough (roughly corresponds the continuous formation of filopodia and the fact that large spines persist longer). Machine learning at the moment is an empirical science that has made great strides without consulting neuroscience for it. I think that has been a good thing: without having to bend towars some biological plausibility researchers have been more exploratory and creative, which has led to the creation of an empirical body of knowledge from which neuroscience could benefit in the future. OTOH, having watched the field of computational neuroscience there has not been a lot of progress since , basically the 80s. So i believe it would be best to leave each of the two fields go their own way.

trextrex 9y ago
> OTOH, having watched the field of computational neuroscience there has not been a lot of progress since , basically the 80s. So i believe it would be best to leave each of the two fields go their own way. I wouldn't really say that. There's a lot of progress being made in using and understanding biological processes for useful tasks. Not surprisingly, the biological mechanisms are surprisingly complex and rich and vary a lot through the brain (like you mentioned with the dendrites. Dendrites also work like coincidence detection mechanisms sometimes in some layers of the cortex [1] for instance) [2] gives a very nice overview of what is needed from both machine learning and computational neuroscience to solve the entire problem of understanding the human brain. There was the liquid state computing paper [3] in 2002 which showed that random networks of spiking neurons can perform some computations and have memory even in the absence of special learning rules. There has also been quite some work on understanding e.g. the role of assemblies of neurons (groups of neurons firing), assembly sequences, plasticity rules, theory formulating neural activity in a probabilistic manner, a better understanding of the role of inhibition, dendrites etc. And lastly, there have been huge advances in neuromorphic computing, both digital and analog. In some cases, the performance on these chips (that use spiking neurons) approaches that of state of the art machine learning. e.g. [4] [1] http://www.sciencedirect.com/science/article/pii/S0166223612002032 http://www.sciencedirect.com/science/article/pii/S0166223612... [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ [3] http://www.mitpressjournals.org/doi/abs/10.1162/089976602760407955 http://www.mitpressjournals.org/doi/abs/10.1162/089976602760... [4] http://www.pnas.org/content/113/41/11441.abstract http://www.pnas.org/content/113/41/11441.abstract
- Seanny123 9y ago
  Nice references! I have some criticism of "Toward an Integration of Deep Learning and Neuroscience" which I've talked to the authors about. I felt like they downplayed the importance of integration between neuroscience and machine learning [1], while over-playing the biological plausibility of Deep Learning [2]. I'm also uncomfortable with the characterization of Neuromorphic computing as "Deep Learning on a Chip", as this dismisses the possibility of online learning and neural dynamic systems [3], which I think are necessary for truly intelligent systems. As for Liquid State Machines, those evolved into Reservoir Computing and Echo State Machines. If you want to read more about them, I would recommend this paper comparing them to the Neural Engineering Framework [4] (with code!) to get a good idea of the state of the field. [1] https://medium.com/@seanaubin/deep-learning-is-almost-the-brain-3aaecd924f3d https://medium.com/@seanaubin/deep-learning-is-almost-the-br... [2] https://cogsci.stackexchange.com/q/16269/4397 https://cogsci.stackexchange.com/q/16269/4397 [3] https://medium.com/@seanaubin/a-way-around-the-coming-performance-walls-neuromorphic-hardware-with-spiking-neurons-facd4291b201 https://medium.com/@seanaubin/a-way-around-the-coming-perfor... [4] https://github.com/arvoelke/delay2017/blob/master/delay2017.pdf https://github.com/arvoelke/delay2017/blob/master/delay2017....
- return0 9y ago
  Definitely people are working, but we dont have an addition to our methods of something fundamental. The papers you re referencing provide interesting conceptual theories, but none of them will survive for decades the way Hodkin-huxley's or Rall's models did. A lot of work on dendrites goes back to the 90s (e.g. seminal work of Mel http://www.nature.com/neuro/journal/v7/n6/abs/nn1253.html http://www.nature.com/neuro/journal/v7/n6/abs/nn1253.html) and we still dont have a convincing theory of what they do.
closed 9y ago
> Machine learning at the moment is an empirical science that has made great strides without consulting neuroscience for it. What is your basis for this statement? Consider Geoff Hinton's publishing record, which contains many collaborations with notable psychologists / neuroscientists (e.g. Jay McClelland) who helped bring neural networks back into the spotlight. http://www.cs.toronto.edu/~hinton/papers.html http://www.cs.toronto.edu/~hinton/papers.html
- mindcrime 9y ago
  True. But, conversely, Hinton himself says in some of his videos (sorry, don't have the specific link in front of me) something to the effect of "ANN's are only very loosely modeled on real neurons and we make no pretense that this is anything like the way a real brain works". (Paraphrased, possibly poorly). The sense I get is that Machine Learning is mostly an empirical field at the moment, without a terribly solid theoretical underpinning. Which is not, of course, to suggest that ML researchers haven't consulted neuroscience at all. But there still seems to be a pretty big disconnect between neuroscience and ML. Unless I've just really missed something, which is entirely possible.
  - closed 9y ago
    Yeah, that's fair. The goals of machine learning and neuroscience seem fairly different. Nobody in ML will complain if your neural network has great prediction but isn't supported biologically. OTOH, neuroscientists might take a model that isn't the best in terms of prediction, if it seems to better represents how the brain operates. There seems to be some interesting interplay, though. For example deepmind recently sponsored this ANN conference where most the speakers were neuroscientists. https://sites.google.com/site/ncpw15/ https://sites.google.com/site/ncpw15/
- return0 9y ago
  I think its fair to say that Hinton's justifications are post-hoc. I think a lot of neural network scientists also do that, as there are many basic components that dont have a biological analogue or even contradict biology, like backpropagation, LSTMs, RBM training layer-by-layer, semilinear activation functions. I can't think of an instance for example where the designers made a choice of connectivity "because that is how the brain does it" and they ended up having a better performing network. There have been recently attempts to match ANNs with spiking networks (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/), but only rough sketches of ideas are provided.
Seanny123 9y ago
> I was disappointed to find out that numenta does not really have collaborations with neuroscientists to test or adapt their theories. Their theories in general are not well known to computational neuroscientists either. In that sense, i m not even sure about the authority of numenta on neocortical theories. Numenta isn't biologically plausible [1], so although the resulting networks might be mappable to the brain in the same manner as Deep Learning and FORCE trained networks, there's no way to map the learning period onto brains. [1] https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef-spa/257/12?u=seanny123 https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef...
rhyolight_ 9y ago
Hi, this is Matt Taylor from Numenta. What I said, exactly, is "We have relationships with some neuroscientists". You can see the text (https://www.reddit.com/r/artificial/comments/6beeqj/5182017_1200_pm_pst_iama_with_matt_taylor_numenta/dho309s/ https://www.reddit.com/r/artificial/comments/6beeqj/5182017_...) and a video of me talking during the AMA about our relationships with neuroscientists (https://youtu.be/9fk8tg_jqh0?t=43m6s https://youtu.be/9fk8tg_jqh0?t=43m6s). The fact is that we have many ongoing relationships with neuroscientists. But I don't know them personally so I didn't give any details. We also have a Guest Scientist program aimed at increasing these collaborations.

trextrex 9y ago

> OTOH, having watched the field of computational neuroscience there has not been a lot of progress since , basically the 80s. So i believe it would be best to leave each of the two fields go their own way. I wouldn't really say that. There's a lot of progress being made in using and understanding biological processes for useful tasks. Not surprisingly, the biological mechanisms are surprisingly complex and rich and vary a lot through the brain (like you mentioned with the dendrites. Dendrites also work like coincidence detection mechanisms sometimes in some layers of the cortex [1] for instance) [2] gives a very nice overview of what is needed from both machine learning and computational neuroscience to solve the entire problem of understanding the human brain. There was the liquid state computing paper [3] in 2002 which showed that random networks of spiking neurons can perform some computations and have memory even in the absence of special learning rules. There has also been quite some work on understanding e.g. the role of assemblies of neurons (groups of neurons firing), assembly sequences, plasticity rules, theory formulating neural activity in a probabilistic manner, a better understanding of the role of inhibition, dendrites etc. And lastly, there have been huge advances in neuromorphic computing, both digital and analog. In some cases, the performance on these chips (that use spiking neurons) approaches that of state of the art machine learning. e.g. [4] [1] http://www.sciencedirect.com/science/article/pii/S0166223612002032 http://www.sciencedirect.com/science/article/pii/S0166223612... [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ [3] http://www.mitpressjournals.org/doi/abs/10.1162/089976602760407955 http://www.mitpressjournals.org/doi/abs/10.1162/089976602760... [4] http://www.pnas.org/content/113/41/11441.abstract http://www.pnas.org/content/113/41/11441.abstract

Seanny123 9y ago
Nice references! I have some criticism of "Toward an Integration of Deep Learning and Neuroscience" which I've talked to the authors about. I felt like they downplayed the importance of integration between neuroscience and machine learning [1], while over-playing the biological plausibility of Deep Learning [2]. I'm also uncomfortable with the characterization of Neuromorphic computing as "Deep Learning on a Chip", as this dismisses the possibility of online learning and neural dynamic systems [3], which I think are necessary for truly intelligent systems. As for Liquid State Machines, those evolved into Reservoir Computing and Echo State Machines. If you want to read more about them, I would recommend this paper comparing them to the Neural Engineering Framework [4] (with code!) to get a good idea of the state of the field. [1] https://medium.com/@seanaubin/deep-learning-is-almost-the-brain-3aaecd924f3d https://medium.com/@seanaubin/deep-learning-is-almost-the-br... [2] https://cogsci.stackexchange.com/q/16269/4397 https://cogsci.stackexchange.com/q/16269/4397 [3] https://medium.com/@seanaubin/a-way-around-the-coming-performance-walls-neuromorphic-hardware-with-spiking-neurons-facd4291b201 https://medium.com/@seanaubin/a-way-around-the-coming-perfor... [4] https://github.com/arvoelke/delay2017/blob/master/delay2017.pdf https://github.com/arvoelke/delay2017/blob/master/delay2017....
return0 9y ago
Definitely people are working, but we dont have an addition to our methods of something fundamental. The papers you re referencing provide interesting conceptual theories, but none of them will survive for decades the way Hodkin-huxley's or Rall's models did. A lot of work on dendrites goes back to the 90s (e.g. seminal work of Mel http://www.nature.com/neuro/journal/v7/n6/abs/nn1253.html http://www.nature.com/neuro/journal/v7/n6/abs/nn1253.html) and we still dont have a convincing theory of what they do.

closed 9y ago

> Machine learning at the moment is an empirical science that has made great strides without consulting neuroscience for it. What is your basis for this statement? Consider Geoff Hinton's publishing record, which contains many collaborations with notable psychologists / neuroscientists (e.g. Jay McClelland) who helped bring neural networks back into the spotlight. http://www.cs.toronto.edu/~hinton/papers.html http://www.cs.toronto.edu/~hinton/papers.html

mindcrime 9y ago
True. But, conversely, Hinton himself says in some of his videos (sorry, don't have the specific link in front of me) something to the effect of "ANN's are only very loosely modeled on real neurons and we make no pretense that this is anything like the way a real brain works". (Paraphrased, possibly poorly). The sense I get is that Machine Learning is mostly an empirical field at the moment, without a terribly solid theoretical underpinning. Which is not, of course, to suggest that ML researchers haven't consulted neuroscience at all. But there still seems to be a pretty big disconnect between neuroscience and ML. Unless I've just really missed something, which is entirely possible.
- closed 9y ago
  Yeah, that's fair. The goals of machine learning and neuroscience seem fairly different. Nobody in ML will complain if your neural network has great prediction but isn't supported biologically. OTOH, neuroscientists might take a model that isn't the best in terms of prediction, if it seems to better represents how the brain operates. There seems to be some interesting interplay, though. For example deepmind recently sponsored this ANN conference where most the speakers were neuroscientists. https://sites.google.com/site/ncpw15/ https://sites.google.com/site/ncpw15/
return0 9y ago
I think its fair to say that Hinton's justifications are post-hoc. I think a lot of neural network scientists also do that, as there are many basic components that dont have a biological analogue or even contradict biology, like backpropagation, LSTMs, RBM training layer-by-layer, semilinear activation functions. I can't think of an instance for example where the designers made a choice of connectivity "because that is how the brain does it" and they ended up having a better performing network. There have been recently attempts to match ANNs with spiking networks (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5021692/), but only rough sketches of ideas are provided.

Seanny123 9y ago

> I was disappointed to find out that numenta does not really have collaborations with neuroscientists to test or adapt their theories. Their theories in general are not well known to computational neuroscientists either. In that sense, i m not even sure about the authority of numenta on neocortical theories. Numenta isn't biologically plausible [1], so although the resulting networks might be mappable to the brain in the same manner as Deep Learning and FORCE trained networks, there's no way to map the learning period onto brains. [1] https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef-spa/257/12?u=seanny123 https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef...

rhyolight_ 9y ago

Hi, this is Matt Taylor from Numenta. What I said, exactly, is "We have relationships with some neuroscientists". You can see the text (https://www.reddit.com/r/artificial/comments/6beeqj/5182017_1200_pm_pst_iama_with_matt_taylor_numenta/dho309s/ https://www.reddit.com/r/artificial/comments/6beeqj/5182017_...) and a video of me talking during the AMA about our relationships with neuroscientists (https://youtu.be/9fk8tg_jqh0?t=43m6s https://youtu.be/9fk8tg_jqh0?t=43m6s). The fact is that we have many ongoing relationships with neuroscientists. But I don't know them personally so I didn't give any details. We also have a Guest Scientist program aimed at increasing these collaborations.