4 ms·

"Neuron Bursts Can Mimic Famous AI Learning Strategy" is an obnoxiously ass backwards headline given that techniques used in AI, such as back propagation, are t

"Neuron Bursts Can Mimic Famous AI Learning Strategy" is an obnoxiously ass backwards headline given that techniques used in AI, such as back propagation, are to some extent modelled on the behaviour of real neurons. AI, and neural networks in particular, are heavily influenced by the behaviour of real neurons and real brains so it's hardly news when brains do things that "mimic" AI. Geez.

FeepingCreature 5y ago
To my layman knowledge, backpropagation was never modelled on real neurons, if only because we didn't think that neurons could do it. At any rate, the degree to which neural networks are modelled on neurons, despite the name, is vastly overstated. A neuron and a ReLU are very different beasts.
- dnautics 5y ago
  the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No evidence that either were faithful models, if my history was correct. ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.
  - soraki_soladead 5y ago
    Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.
    - dnautics 5y ago
      > spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
  - mjburgess 5y ago
    It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
    - dnautics 5y ago
      > its somewhat incoherent to suppose it does Agreed, but the principle of convergent evolution is a thing. It's possible that and we cannot rule out that we are accidentally stumbling our way into homologous models of biological compute (I personally doubt it).
      - mjburgess 5y ago
        Well in the sense that the activation map over the brain is a function of space-time, ie., A(x, t) -- it is possible that the activations in a NN might instantiate this function... but that's literally probability 0. As the set of such functions, R^2 -> R are uncountable infinite.
        dnautics 5y ago
        If you imagine the space of all possibilities as R^n -> R and the solution to be only one mapping function and not "some subspace of the possibility space", possibly around some ball of results, I got some news for you.
- mjburgess 5y ago
  Yes, indeed. There is no such thing as "backpropagation" in the brain. The modern algorithm called "Neural Networks" has nothing to do with the 1940s model of the brain called the "Neural Network" model. The modern algorithm is a supervised optimisation process which approximates functions using a piece-wise linear regression model. Nothing about that algorithm, nor the function-approximations it produces, is based on anything in the brain. It's a bit like saying the Tesla was inspired by horses.
  - Retric 5y ago
    The 1940s model didn’t include any way to generate them, so talking about their creation process is kind of meaningless. The process of generating a NN is it’s own thing, just as python code is different from a python programmer.
neurobot123 5y ago
The whole study is an like a hammer looking for a nail, there is no evidence for anything like backprop, “credit assignment” or reinforcement learning in the human or other animal brain. This would require some kind of global “teaching signal” and iterative optimization which is not biologically plausible.
soraki_soladead 5y ago
This is a poor hot take. The title isn’t sensationalized. It’s pretty accurate. Neural networks are only in the loosest sense inspired by biology. Backprop itself was not inspired by biology. The implementation of backprop has been known for a while to not be biologically plausible for a number of reasons, some covered in the article. Even the paper’s burst method isn’t backprop. It’s a mechanism for error transport. Errors themselves are still localized which is a big break from backprop. Thus “mimicking” is correct because only the outcome is similar, not the mechanism.

FeepingCreature 5y ago

To my layman knowledge, backpropagation was never modelled on real neurons, if only because we didn't think that neurons could do it. At any rate, the degree to which neural networks are modelled on neurons, despite the name, is vastly overstated. A neuron and a ReLU are very different beasts.

dnautics 5y ago
the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No evidence that either were faithful models, if my history was correct. ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.
- soraki_soladead 5y ago
  Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.
  - dnautics 5y ago
    > spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
- mjburgess 5y ago
  It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
  - dnautics 5y ago
    > its somewhat incoherent to suppose it does Agreed, but the principle of convergent evolution is a thing. It's possible that and we cannot rule out that we are accidentally stumbling our way into homologous models of biological compute (I personally doubt it).
    - mjburgess 5y ago
      Well in the sense that the activation map over the brain is a function of space-time, ie., A(x, t) -- it is possible that the activations in a NN might instantiate this function... but that's literally probability 0. As the set of such functions, R^2 -> R are uncountable infinite.
      - dnautics 5y ago
        If you imagine the space of all possibilities as R^n -> R and the solution to be only one mapping function and not "some subspace of the possibility space", possibly around some ball of results, I got some news for you.
mjburgess 5y ago
Yes, indeed. There is no such thing as "backpropagation" in the brain. The modern algorithm called "Neural Networks" has nothing to do with the 1940s model of the brain called the "Neural Network" model. The modern algorithm is a supervised optimisation process which approximates functions using a piece-wise linear regression model. Nothing about that algorithm, nor the function-approximations it produces, is based on anything in the brain. It's a bit like saying the Tesla was inspired by horses.
- Retric 5y ago
  The 1940s model didn’t include any way to generate them, so talking about their creation process is kind of meaningless. The process of generating a NN is it’s own thing, just as python code is different from a python programmer.

neurobot123 5y ago

The whole study is an like a hammer looking for a nail, there is no evidence for anything like backprop, “credit assignment” or reinforcement learning in the human or other animal brain. This would require some kind of global “teaching signal” and iterative optimization which is not biologically plausible.

soraki_soladead 5y ago

This is a poor hot take. The title isn’t sensationalized. It’s pretty accurate. Neural networks are only in the loosest sense inspired by biology. Backprop itself was not inspired by biology. The implementation of backprop has been known for a while to not be biologically plausible for a number of reasons, some covered in the article. Even the paper’s burst method isn’t backprop. It’s a mechanism for error transport. Errors themselves are still localized which is a big break from backprop. Thus “mimicking” is correct because only the outcome is similar, not the mechanism.