5 ms·

I honestly think that focusing on asynchronicity is almost completely the wrong way to go about designing neuromorphic systems. One thing that engineers almost

I honestly think that focusing on asynchronicity is almost completely the wrong way to go about designing neuromorphic systems. One thing that engineers almost never focus on, but is a well-researched vital aspect of the brain, is the concept of firing frequency and synchronicity: "cells that fire together wire together". This aspect of synchronized, timed firing is so strong that it creates neural oscillations on a large scale that are consistently characteristic of different brain states (brain waves) and widely theorized to be important to how information is encoded in the brain. It's also very commonly observed that, for instance, the intensity of a stimulus is encoded not in the amplitude of a particular neuron's response but rather in the frequency of a neuron's firing. This oscillatory activity has been shown to be active at all different levels of organization, too. There's basically an entire dimension to how the brain works (in terms of positively and negatively interfering wave patterns localized in different parts of the brain, with neurons that serve to speed up or slow down a continuous pattern of firing) that people looking to reproduce its function usually totally ignore. They think of neuronal firings as discrete calculations, when really they're transformations on an ongoing pattern of activity that's evolving over time. I feel this way of looking at the brain is far more consistent with how biological processes generally function: the point is not that each neuron is accomplishing some precise operation independent of the other neurons, but rather that with a large enough population of imprecise neurons firing individually an overall pattern of useful activity emerges.

ianai 9y ago
I think directly emulating the current understanding of biological systems isn’t necessarily the best way. It’s a losing proposition to emulate hardware with other hardware that’s fundamentally different from the original. I think the better way would be emulating what we understand of the ‘software’: memory, concepts, language/communication, etc. (I say that all with great humility. This is by no means a field I can speak about with authority.
- aalleavitch 9y ago
  That's the problem, though: the idea behind a neuromorphic system is that it works like the brain, but what these "neuromorphic" chips do relies on an extremely vague relationship between a type of computing we know how to do (parallel processing), and what the brain actually does. Furthermore, I think that the type of deep learning that actually works well (like recurrent neural nets) relies more on these aspects of the brain (synchronized and oscillatory activity) than a lot of engineers working in the field realize as they chase after other concepts of how neurons might work. I feel like there's a lot of breakthroughs that could be made in the field of neuromorphic computing (and honestly in our understanding of how the brain works) if there were simply more people who were equally well-trained enough in neuroscience and computing to see the deeper relationships. As it is, I feel like the majority of experts that have deep knowledge in one of these fields are hampered by a superficial understanding of what's going on in the other. We're really only scratching the surface of how we could design a system that approximates what the brain does.
  - arketyp 9y ago
    Out of curiousity given your seemingly strong sentiments, what is your background and what authors/researchers would you say have influenced your thinking the most?
    - aalleavitch 9y ago
      I'm just a frontend engineer who dabbles in ML and cognitive science; my neuroscience experience is limited to a bachelor's degree and a couple years of lab work so I'm admittedly a lot more passionate than I am personally authoritative on the subject. The biggest influence on my thinking regarding intelligent systems generally is Douglas Hofstadter, but the notions of brain function I alluded to above are prevalent throughout a lot of computational neuroscience.
      - arketyp 9y ago
        I sympathize with this being in a similar professional situation and having once been introduced to this field via Hofstadter. His musings are usually on a distinctly symbolic level however, and though it is not my intention to defend a kind of results-only pragmatic approach, I do concern myself with strategies for progressively reaching better models of intelligence and their realization in contemporary hardware in particular. What, for instance, is your opinion about Hinton's recently published capsules idea? https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 https://openreview.net/forum?id=HJWLfGWRb https://openreview.net/forum?id=HJWLfGWRb
shahbaby 9y ago
This is like saying that all flying birds have feathers therefore we must have something similar to feathers on a flying machine. Simply emulating things just because it feels like they should be important is not a particularly promising approach when you're designing circuits.
- aalleavitch 9y ago
  To torture your analogy: in my mind, if the brain is like a bird, then modern approaches to machine learning are like helicopters. Yes, these things can both fly, but they don't fly in nearly the same way, and if your helicopter isn't getting off the ground then looking at the design of a bird isn't going to help you very much with many of the challenges you're going to face in getting it to work. You wouldn't claim that a helicopter is bird-shaped, even though there's some basic relationships between the aerodynamic principles of a bird's wings and a helicopter rotor. If we're having trouble getting our helicopter to work, maybe we should be trying to make a working airplane first, since then we can base more of it off the design of a bird and use this to help us to better understand the basic principles of aerodynamics. Do you get where I'm coming from here?
  - shahbaby 9y ago
    To be clear, once we understand the principles behind the neocortex, only then it will make sense to ask questions about the importance of oscillations or how it relates to asynchronous vs synchronous systems, and many more phenomenon we have observed but do not really understand. Without an underlying theory to tie it all together, it's difficult to make sense of it. I personally think that oscillations are more likely to be an emergent property which is a common theme in nature.
- rdlecler1 9y ago
  Both are operating according to principles of thermodynamics. That’s what we lack and that’s what a biologically defensible model of AI can teach us.
dkersten 9y ago
> It's also very commonly observed that, for instance, the intensity of a stimulus is encoded not in the amplitude of a particular neuron's response but rather in the frequency of a neuron's firing. This was actually mentioned by Geoff Hinton in his deep learning coursera lectures and the reason why his RBM's were outputting binary signals rather than float values -- the timing was more important than the values. It made a lot of sense when he talked about it. I saw this back in 2012. I assumed this was commonly understood now, but I guess not?
- arca_vorago 9y ago
  I've hypothesised the optimum way would be to use both binary and floats at the same time after a paper I read that basically said neuron pathways are both digital and analog. Something about the analog (float) being a weight pathway indicator or something. I wish I knew more about neuroscience.
  - dkersten 9y ago
    hmm interesting. I wonder where I could find out more.
crusso 9y ago
Caveat: I'm jumping out of my depth and just throwing my non-neural-researcher understanding out there. At the individual neuron level, everything is asynchronous, right? The firing/pulsing behaviors happen at a network level and this implementation should still see harmonic effects. In fact, you might even see richer harmonic effects because the neurons are asynchronous and not all clocked together. So while you're probably right that firing/pulsing/synchronicity behaviors are a vital aspect of the brain, at the network level the best way for us to see more brain-like harmonics will be to have asynchronous neurons.
- vorotato 9y ago
  yeah logically it seems like it would be impossible to have patterns of synchronicity without asynchronous neurons.
aaavl2821 9y ago
I agree that most computer scientists working on anything "neural" would benefit from a more biology-based understanding of the brain. But I think the article was using "synchronicity" in a different way, in that the bad thing about synchronous hardware is that it forces all computations to fit into a set time period, even though computations take less time. So removing that barrier is an efficiency increase. And I think the "fire together wire together" thing is more about synaptic plasticity and doesn't really relate to the concept of synchronicity as described in the article (to my understanding). Fire together wire together means When a neuron fires as a result of signal from another one, certain molecules are created to strengthen the neurons relationship. There is nothing timing / synchronous about this, using timing in the sense of requiring a third-party "scheduler" to regulate firing. Individual neurons are not regulated by this constraint. But yes, there are so many other amazing features of neurons that we don't fully understand and that aren't used in computer design because of a lack of basic knowledge about the biology and chemistry.

ianai 9y ago

I think directly emulating the current understanding of biological systems isn’t necessarily the best way. It’s a losing proposition to emulate hardware with other hardware that’s fundamentally different from the original. I think the better way would be emulating what we understand of the ‘software’: memory, concepts, language/communication, etc. (I say that all with great humility. This is by no means a field I can speak about with authority.

aalleavitch 9y ago
That's the problem, though: the idea behind a neuromorphic system is that it works like the brain, but what these "neuromorphic" chips do relies on an extremely vague relationship between a type of computing we know how to do (parallel processing), and what the brain actually does. Furthermore, I think that the type of deep learning that actually works well (like recurrent neural nets) relies more on these aspects of the brain (synchronized and oscillatory activity) than a lot of engineers working in the field realize as they chase after other concepts of how neurons might work. I feel like there's a lot of breakthroughs that could be made in the field of neuromorphic computing (and honestly in our understanding of how the brain works) if there were simply more people who were equally well-trained enough in neuroscience and computing to see the deeper relationships. As it is, I feel like the majority of experts that have deep knowledge in one of these fields are hampered by a superficial understanding of what's going on in the other. We're really only scratching the surface of how we could design a system that approximates what the brain does.
- arketyp 9y ago
  Out of curiousity given your seemingly strong sentiments, what is your background and what authors/researchers would you say have influenced your thinking the most?
  - aalleavitch 9y ago
    I'm just a frontend engineer who dabbles in ML and cognitive science; my neuroscience experience is limited to a bachelor's degree and a couple years of lab work so I'm admittedly a lot more passionate than I am personally authoritative on the subject. The biggest influence on my thinking regarding intelligent systems generally is Douglas Hofstadter, but the notions of brain function I alluded to above are prevalent throughout a lot of computational neuroscience.
    - arketyp 9y ago
      I sympathize with this being in a similar professional situation and having once been introduced to this field via Hofstadter. His musings are usually on a distinctly symbolic level however, and though it is not my intention to defend a kind of results-only pragmatic approach, I do concern myself with strategies for progressively reaching better models of intelligence and their realization in contemporary hardware in particular. What, for instance, is your opinion about Hinton's recently published capsules idea? https://arxiv.org/abs/1710.09829 https://arxiv.org/abs/1710.09829 https://openreview.net/forum?id=HJWLfGWRb https://openreview.net/forum?id=HJWLfGWRb

shahbaby 9y ago

This is like saying that all flying birds have feathers therefore we must have something similar to feathers on a flying machine. Simply emulating things just because it feels like they should be important is not a particularly promising approach when you're designing circuits.

aalleavitch 9y ago
To torture your analogy: in my mind, if the brain is like a bird, then modern approaches to machine learning are like helicopters. Yes, these things can both fly, but they don't fly in nearly the same way, and if your helicopter isn't getting off the ground then looking at the design of a bird isn't going to help you very much with many of the challenges you're going to face in getting it to work. You wouldn't claim that a helicopter is bird-shaped, even though there's some basic relationships between the aerodynamic principles of a bird's wings and a helicopter rotor. If we're having trouble getting our helicopter to work, maybe we should be trying to make a working airplane first, since then we can base more of it off the design of a bird and use this to help us to better understand the basic principles of aerodynamics. Do you get where I'm coming from here?
- shahbaby 9y ago
  To be clear, once we understand the principles behind the neocortex, only then it will make sense to ask questions about the importance of oscillations or how it relates to asynchronous vs synchronous systems, and many more phenomenon we have observed but do not really understand. Without an underlying theory to tie it all together, it's difficult to make sense of it. I personally think that oscillations are more likely to be an emergent property which is a common theme in nature.
rdlecler1 9y ago
Both are operating according to principles of thermodynamics. That’s what we lack and that’s what a biologically defensible model of AI can teach us.

dkersten 9y ago

> It's also very commonly observed that, for instance, the intensity of a stimulus is encoded not in the amplitude of a particular neuron's response but rather in the frequency of a neuron's firing. This was actually mentioned by Geoff Hinton in his deep learning coursera lectures and the reason why his RBM's were outputting binary signals rather than float values -- the timing was more important than the values. It made a lot of sense when he talked about it. I saw this back in 2012. I assumed this was commonly understood now, but I guess not?

arca_vorago 9y ago
I've hypothesised the optimum way would be to use both binary and floats at the same time after a paper I read that basically said neuron pathways are both digital and analog. Something about the analog (float) being a weight pathway indicator or something. I wish I knew more about neuroscience.
- dkersten 9y ago
  hmm interesting. I wonder where I could find out more.

crusso 9y ago

Caveat: I'm jumping out of my depth and just throwing my non-neural-researcher understanding out there. At the individual neuron level, everything is asynchronous, right? The firing/pulsing behaviors happen at a network level and this implementation should still see harmonic effects. In fact, you might even see richer harmonic effects because the neurons are asynchronous and not all clocked together. So while you're probably right that firing/pulsing/synchronicity behaviors are a vital aspect of the brain, at the network level the best way for us to see more brain-like harmonics will be to have asynchronous neurons.

vorotato 9y ago
yeah logically it seems like it would be impossible to have patterns of synchronicity without asynchronous neurons.

aaavl2821 9y ago

I agree that most computer scientists working on anything "neural" would benefit from a more biology-based understanding of the brain. But I think the article was using "synchronicity" in a different way, in that the bad thing about synchronous hardware is that it forces all computations to fit into a set time period, even though computations take less time. So removing that barrier is an efficiency increase. And I think the "fire together wire together" thing is more about synaptic plasticity and doesn't really relate to the concept of synchronicity as described in the article (to my understanding). Fire together wire together means When a neuron fires as a result of signal from another one, certain molecules are created to strengthen the neurons relationship. There is nothing timing / synchronous about this, using timing in the sense of requiring a third-party "scheduler" to regulate firing. Individual neurons are not regulated by this constraint. But yes, there are so many other amazing features of neurons that we don't fully understand and that aren't used in computer design because of a lack of basic knowledge about the biology and chemistry.