4 ms·

Brain bursts can mimic famous AI learning strategy

41 points by sdht0 5y ago

oriel 5y ago
link to paper: https://www.biorxiv.org/content/10.1101/2020.03.30.015511v1 https://www.biorxiv.org/content/10.1101/2020.03.30.015511v1 actual title: Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
echopurity 5y ago
Bottom tier science is top tier HN.
bartread 5y ago
"Neuron Bursts Can Mimic Famous AI Learning Strategy" is an obnoxiously ass backwards headline given that techniques used in AI, such as back propagation, are to some extent modelled on the behaviour of real neurons. AI, and neural networks in particular, are heavily influenced by the behaviour of real neurons and real brains so it's hardly news when brains do things that "mimic" AI. Geez.
- FeepingCreature 5y ago
  To my layman knowledge, backpropagation was never modelled on real neurons, if only because we didn't think that neurons could do it. At any rate, the degree to which neural networks are modelled on neurons, despite the name, is vastly overstated. A neuron and a ReLU are very different beasts.
  - dnautics 5y ago
    the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No evidence that either were faithful models, if my history was correct. ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.
    - soraki_soladead 5y ago
      Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.
      - dnautics 5y ago
        > spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
    - mjburgess 5y ago
      It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
      - dnautics 5y ago
  - mjburgess 5y ago
    Yes, indeed. There is no such thing as "backpropagation" in the brain. The modern algorithm called "Neural Networks" has nothing to do with the 1940s model of the brain called the "Neural Network" model. The modern algorithm is a supervised optimisation process which approximates functions using a piece-wise linear regression model. Nothing about that algorithm, nor the function-approximations it produces, is based on anything in the brain. It's a bit like saying the Tesla was inspired by horses.
    - Retric 5y ago
      The 1940s model didn’t include any way to generate them, so talking about their creation process is kind of meaningless. The process of generating a NN is it’s own thing, just as python code is different from a python programmer.
- neurobot123 5y ago
  The whole study is an like a hammer looking for a nail, there is no evidence for anything like backprop, “credit assignment” or reinforcement learning in the human or other animal brain. This would require some kind of global “teaching signal” and iterative optimization which is not biologically plausible.
- soraki_soladead 5y ago
  This is a poor hot take. The title isn’t sensationalized. It’s pretty accurate. Neural networks are only in the loosest sense inspired by biology. Backprop itself was not inspired by biology. The implementation of backprop has been known for a while to not be biologically plausible for a number of reasons, some covered in the article. Even the paper’s burst method isn’t backprop. It’s a mechanism for error transport. Errors themselves are still localized which is a big break from backprop. Thus “mimicking” is correct because only the outcome is similar, not the mechanism.
MauranKilom 5y ago
> Backpropagation itself is not biologically plausible because, among other things, real neurons can’t just stop processing the external world and wait for backpropagation to begin — if they did, we’d end up with lapses in our vision or hearing. Counterpoint: Our brain works around "lapses in our vision" constantly. See https://en.wikipedia.org/wiki/Chronostasis https://en.wikipedia.org/wiki/Chronostasis or https://en.wikipedia.org/wiki/Blind_spot https://en.wikipedia.org/wiki/Blind_spot. (Not saying that makes backpropagation biologically plausible, but as phrased, their argument is not convincing.)
- mjburgess 5y ago
  The relevant sense of "stop" is stop /processing/. And the relevant object is the neurone, not the perception. ie., our visual field lags during, eg., a saccade; but all the neurones backing the visual system are still responsive to light. And it isnt that the visual field is blank, nor the eye/neurones unresponsive. The neurones are in constant continuous contact with environmental stimuli -- they don't just stop being causally activate.
  - soraki_soladead 5y ago
    Indeed. Neurons don't stop processing during events like saccades. They are hyperpolarized by inhibitory neurons. This suppresses further spikes but it's not a separate mode of operation.
  - MauranKilom 5y ago
    > And the relevant object is the neurone, not the perception. The argument given in the article is very much that you would perceive an interruption if your neurons were momentarily doing something else, but you don't, so they don't ("A -> B, not B, therefore not A"). I'm contesting this implication. You merely presented statements equivalent to "neurons don't do that" ("not A"), which misses the point. Or, phrased another way: How do you know that "[neurons] don't just stop being causally activate"? It is not inherently implausible that, at any given time, some neurons in our brain are "on backpropagation maintenance", so to speak. ANNs can be trained just fine with dropout after all (and for ANNs, we intentionally add dropout because it's beneficial!). I have no evidence for this claim (and I don't believe it to be true in the slightest), but "you'd notice a momentary loss of perception" is simply not an argument against it.
lindseymysse 5y ago
There is an old poem called the Psychomachia by a poet named Prudentious. https://en.wikipedia.org/wiki/Psychomachia https://en.wikipedia.org/wiki/Psychomachia It is a fascinating poem -- it all takes place in the character's head. The virtues and the vices battle it out, individually, but as a complete battle. It is, in my opinion, a great example of a Generated Adversarial Network. The reason a lot of great art ends up representing deep mathematical truths is that art is, at its best, an act of deep observation, whether it is of oneself or the world around oneself. I wouldn't bother reading the Psychomachia -- I translated parts of it in college and I agree with C.S. Lewis that it is a more known poem than a good one. I still found its observation of the mind in parts doing battle useful for myself when I have dealt with mental illness in my life.

41 points by sdht0 5y ago

link to paper: https://www.biorxiv.org/content/10.1101/2020.03.30.015511v1 https://www.biorxiv.org/content/10.1101/2020.03.30.015511v1 actual title: Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits

echopurity 5y ago

Bottom tier science is top tier HN.

bartread 5y ago

"Neuron Bursts Can Mimic Famous AI Learning Strategy" is an obnoxiously ass backwards headline given that techniques used in AI, such as back propagation, are to some extent modelled on the behaviour of real neurons. AI, and neural networks in particular, are heavily influenced by the behaviour of real neurons and real brains so it's hardly news when brains do things that "mimic" AI. Geez.

FeepingCreature 5y ago
To my layman knowledge, backpropagation was never modelled on real neurons, if only because we didn't think that neurons could do it. At any rate, the degree to which neural networks are modelled on neurons, despite the name, is vastly overstated. A neuron and a ReLU are very different beasts.
- dnautics 5y ago
  the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No evidence that either were faithful models, if my history was correct. ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.
  - soraki_soladead 5y ago
    Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.
    - dnautics 5y ago
      > spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
  - mjburgess 5y ago
    It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
    - dnautics 5y ago
- mjburgess 5y ago
  Yes, indeed. There is no such thing as "backpropagation" in the brain. The modern algorithm called "Neural Networks" has nothing to do with the 1940s model of the brain called the "Neural Network" model. The modern algorithm is a supervised optimisation process which approximates functions using a piece-wise linear regression model. Nothing about that algorithm, nor the function-approximations it produces, is based on anything in the brain. It's a bit like saying the Tesla was inspired by horses.
  - Retric 5y ago
    The 1940s model didn’t include any way to generate them, so talking about their creation process is kind of meaningless. The process of generating a NN is it’s own thing, just as python code is different from a python programmer.
neurobot123 5y ago
The whole study is an like a hammer looking for a nail, there is no evidence for anything like backprop, “credit assignment” or reinforcement learning in the human or other animal brain. This would require some kind of global “teaching signal” and iterative optimization which is not biologically plausible.
soraki_soladead 5y ago
This is a poor hot take. The title isn’t sensationalized. It’s pretty accurate. Neural networks are only in the loosest sense inspired by biology. Backprop itself was not inspired by biology. The implementation of backprop has been known for a while to not be biologically plausible for a number of reasons, some covered in the article. Even the paper’s burst method isn’t backprop. It’s a mechanism for error transport. Errors themselves are still localized which is a big break from backprop. Thus “mimicking” is correct because only the outcome is similar, not the mechanism.

MauranKilom 5y ago

> Backpropagation itself is not biologically plausible because, among other things, real neurons can’t just stop processing the external world and wait for backpropagation to begin — if they did, we’d end up with lapses in our vision or hearing. Counterpoint: Our brain works around "lapses in our vision" constantly. See https://en.wikipedia.org/wiki/Chronostasis https://en.wikipedia.org/wiki/Chronostasis or https://en.wikipedia.org/wiki/Blind_spot https://en.wikipedia.org/wiki/Blind_spot. (Not saying that makes backpropagation biologically plausible, but as phrased, their argument is not convincing.)

mjburgess 5y ago
The relevant sense of "stop" is stop /processing/. And the relevant object is the neurone, not the perception. ie., our visual field lags during, eg., a saccade; but all the neurones backing the visual system are still responsive to light. And it isnt that the visual field is blank, nor the eye/neurones unresponsive. The neurones are in constant continuous contact with environmental stimuli -- they don't just stop being causally activate.
- soraki_soladead 5y ago
  Indeed. Neurons don't stop processing during events like saccades. They are hyperpolarized by inhibitory neurons. This suppresses further spikes but it's not a separate mode of operation.
- MauranKilom 5y ago
  > And the relevant object is the neurone, not the perception. The argument given in the article is very much that you would perceive an interruption if your neurons were momentarily doing something else, but you don't, so they don't ("A -> B, not B, therefore not A"). I'm contesting this implication. You merely presented statements equivalent to "neurons don't do that" ("not A"), which misses the point. Or, phrased another way: How do you know that "[neurons] don't just stop being causally activate"? It is not inherently implausible that, at any given time, some neurons in our brain are "on backpropagation maintenance", so to speak. ANNs can be trained just fine with dropout after all (and for ANNs, we intentionally add dropout because it's beneficial!). I have no evidence for this claim (and I don't believe it to be true in the slightest), but "you'd notice a momentary loss of perception" is simply not an argument against it.

lindseymysse 5y ago

There is an old poem called the Psychomachia by a poet named Prudentious. https://en.wikipedia.org/wiki/Psychomachia https://en.wikipedia.org/wiki/Psychomachia It is a fascinating poem -- it all takes place in the character's head. The virtues and the vices battle it out, individually, but as a complete battle. It is, in my opinion, a great example of a Generated Adversarial Network. The reason a lot of great art ends up representing deep mathematical truths is that art is, at its best, an act of deep observation, whether it is of oneself or the world around oneself. I wouldn't bother reading the Psychomachia -- I translated parts of it in college and I agree with C.S. Lewis that it is a more known poem than a good one. I still found its observation of the mind in parts doing battle useful for myself when I have dealt with mental illness in my life.