4 ms·

That's the problem, though: the idea behind a neuromorphic system is that it works like the brain, but what these "neuromorphic" chips do relies on an extremely

That's the problem, though: the idea behind a neuromorphic system is that it works like the brain, but what these "neuromorphic" chips do relies on an extremely vague relationship between a type of computing we know how to do (parallel processing), and what the brain actually does. Furthermore, I think that the type of deep learning that actually works well (like recurrent neural nets) relies more on these aspects of the brain (synchronized and oscillatory activity) than a lot of engineers working in the field realize as they chase after other concepts of how neurons might work. I feel like there's a lot of breakthroughs that could be made in the field of neuromorphic computing (and honestly in our understanding of how the brain works) if there were simply more people who were equally well-trained enough in neuroscience and computing to see the deeper relationships. As it is, I feel like the majority of experts that have deep knowledge in one of these fields are hampered by a superficial understanding of what's going on in the other. We're really only scratching the surface of how we could design a system that approximates what the brain does.

arketyp 9y ago
Out of curiousity given your seemingly strong sentiments, what is your background and what authors/researchers would you say have influenced your thinking the most?
- aalleavitch 9y ago
  I'm just a frontend engineer who dabbles in ML and cognitive science; my neuroscience experience is limited to a bachelor's degree and a couple years of lab work so I'm admittedly a lot more passionate than I am personally authoritative on the subject. The biggest influence on my thinking regarding intelligent systems generally is Douglas Hofstadter, but the notions of brain function I alluded to above are prevalent throughout a lot of computational neuroscience.
  - arketyp 9y ago
    I sympathize with this being in a similar professional situation and having once been introduced to this field via Hofstadter. His musings are usually on a distinctly symbolic level however, and though it is not my intention to defend a kind of results-only pragmatic approach, I do concern myself with strategies for progressively reaching better models of intelligence and their realization in contemporary hardware in particular. What, for instance, is your opinion about Hinton's recently published capsules idea? https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 https://openreview.net/forum?id=HJWLfGWRb https://openreview.net/forum?id=HJWLfGWRb
    - aalleavitch 9y ago
      I haven't had the opportunity to dive deeply into capsule networks yet, but I have been fairly excited by what little I've read so far. I think that Hinton et al are brilliant and definitely taking neural nets in the right direction by trying to approximate more of the mid-level organization of the brain, but I feel like recent attempts to integrate things like attention and memory into nns are still at a very elementary level. This isn't to denigrate recent work at all: this is a stage we have to cross on our way to more powerful intelligent systems, and you're right that pragmatically one of the best ways to further the field is to focus on making models that are useful in their current state even if they poorly approximate more complex systems. On the whole, I'd wager the state of the field is still at the Bohr model level, but progressing rapidly.
dewhelmed 9y ago
If you don't mind, could you share some resources on existing models that approximate the neuron behaviour more accurately? If you're talking about feedback-driven models like RNNs/LSTMs/GRUs/etc, then I can safely say that there are people working on realising these models on custom computing architectures. The simple feed-forward neural network could be thought of as the low-hanging fruit in this regard, which, nevertheless, is still an impressive feat.
- aalleavitch 9y ago
  This is just a smattering of recent articles from the Journal of Computational Neuroscience, but I think something more along the lines of models like these would give us closer approximations of how neurons really behave: [1] https://link.springer.com/article/10.1007/s10827-017-0667-3 https://link.springer.com/article/10.1007/s10827-017-0667-3 [2] https://link.springer.com/article/10.1007/s10827-017-0668-2 https://link.springer.com/article/10.1007/s10827-017-0668-2 [3] https://link.springer.com/article/10.1007/s10827-017-0655-7 https://link.springer.com/article/10.1007/s10827-017-0655-7 [4] https://link.springer.com/article/10.1007/s10827-017-0646-8 https://link.springer.com/article/10.1007/s10827-017-0646-8 [5] https://link.springer.com/article/10.1007/s10827-017-0658-4 https://link.springer.com/article/10.1007/s10827-017-0658-4
- npatrick04 9y ago
  There's a lot of good information on modeling spiking neurons here: http://www.scholarpedia.org/article/Category:Spiking_Networks http://www.scholarpedia.org/article/Category:Spiking_Network... Also, I like the papers linked here by Eugene Izhikevich: http://www.izhikevich.org/publications/index.htm http://www.izhikevich.org/publications/index.htm The new links discussing the capsule neural networks are somewhat reminiscent of this polychronous computation paper: http://www.izhikevich.org/publications/polychronous_wavefront_computations.htm http://www.izhikevich.org/publications/polychronous_wavefron.... https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...
etiam 9y ago
To compound the problem I think a lot of people suffer from a lack of understanding of their lack of understanding. Being ignorant of what nervous tissue really works like is largely a transitory problem if one stays with it long enough, but "knowing" that it's just like an artificial neural net, or standard electrical circuit design with funky clocking and some memristor gates, is potentially devastating. And if I may be allowed a personal remark, it's refreshing in that context to see your humility about the matter. BSc + some years in the lab may not be a researcher's career but it substantial, and more than most have to show. Liked your comments. I'll be looking for more of them whenever topics like these come up.

arketyp 9y ago

Out of curiousity given your seemingly strong sentiments, what is your background and what authors/researchers would you say have influenced your thinking the most?

aalleavitch 9y ago
I'm just a frontend engineer who dabbles in ML and cognitive science; my neuroscience experience is limited to a bachelor's degree and a couple years of lab work so I'm admittedly a lot more passionate than I am personally authoritative on the subject. The biggest influence on my thinking regarding intelligent systems generally is Douglas Hofstadter, but the notions of brain function I alluded to above are prevalent throughout a lot of computational neuroscience.
- arketyp 9y ago
  I sympathize with this being in a similar professional situation and having once been introduced to this field via Hofstadter. His musings are usually on a distinctly symbolic level however, and though it is not my intention to defend a kind of results-only pragmatic approach, I do concern myself with strategies for progressively reaching better models of intelligence and their realization in contemporary hardware in particular. What, for instance, is your opinion about Hinton's recently published capsules idea? https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 https://openreview.net/forum?id=HJWLfGWRb https://openreview.net/forum?id=HJWLfGWRb
  - aalleavitch 9y ago
    I haven't had the opportunity to dive deeply into capsule networks yet, but I have been fairly excited by what little I've read so far. I think that Hinton et al are brilliant and definitely taking neural nets in the right direction by trying to approximate more of the mid-level organization of the brain, but I feel like recent attempts to integrate things like attention and memory into nns are still at a very elementary level. This isn't to denigrate recent work at all: this is a stage we have to cross on our way to more powerful intelligent systems, and you're right that pragmatically one of the best ways to further the field is to focus on making models that are useful in their current state even if they poorly approximate more complex systems. On the whole, I'd wager the state of the field is still at the Bohr model level, but progressing rapidly.

dewhelmed 9y ago

If you don't mind, could you share some resources on existing models that approximate the neuron behaviour more accurately? If you're talking about feedback-driven models like RNNs/LSTMs/GRUs/etc, then I can safely say that there are people working on realising these models on custom computing architectures. The simple feed-forward neural network could be thought of as the low-hanging fruit in this regard, which, nevertheless, is still an impressive feat.

aalleavitch 9y ago
This is just a smattering of recent articles from the Journal of Computational Neuroscience, but I think something more along the lines of models like these would give us closer approximations of how neurons really behave: [1] https://link.springer.com/article/10.1007/s10827-017-0667-3 https://link.springer.com/article/10.1007/s10827-017-0667-3 [2] https://link.springer.com/article/10.1007/s10827-017-0668-2 https://link.springer.com/article/10.1007/s10827-017-0668-2 [3] https://link.springer.com/article/10.1007/s10827-017-0655-7 https://link.springer.com/article/10.1007/s10827-017-0655-7 [4] https://link.springer.com/article/10.1007/s10827-017-0646-8 https://link.springer.com/article/10.1007/s10827-017-0646-8 [5] https://link.springer.com/article/10.1007/s10827-017-0658-4 https://link.springer.com/article/10.1007/s10827-017-0658-4
npatrick04 9y ago
There's a lot of good information on modeling spiking neurons here: http://www.scholarpedia.org/article/Category:Spiking_Networks http://www.scholarpedia.org/article/Category:Spiking_Network... Also, I like the papers linked here by Eugene Izhikevich: http://www.izhikevich.org/publications/index.htm http://www.izhikevich.org/publications/index.htm The new links discussing the capsule neural networks are somewhat reminiscent of this polychronous computation paper: http://www.izhikevich.org/publications/polychronous_wavefront_computations.htm http://www.izhikevich.org/publications/polychronous_wavefron.... https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...

etiam 9y ago

To compound the problem I think a lot of people suffer from a lack of understanding of their lack of understanding. Being ignorant of what nervous tissue really works like is largely a transitory problem if one stays with it long enough, but "knowing" that it's just like an artificial neural net, or standard electrical circuit design with funky clocking and some memristor gates, is potentially devastating. And if I may be allowed a personal remark, it's refreshing in that context to see your humility about the matter. BSc + some years in the lab may not be a researcher's career but it substantial, and more than most have to show. Liked your comments. I'll be looking for more of them whenever topics like these come up.