5 ms·

the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No ev

the initial idea of a perceptron (circa 1960?) is modeled after neurons, and I think convnets are inspird by what we observed visual cortex architecture. No evidence that either were faithful models, if my history was correct. ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.

soraki_soladead 5y ago
Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.
- dnautics 5y ago
  > spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
mjburgess 5y ago
It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
- dnautics 5y ago
  > its somewhat incoherent to suppose it does Agreed, but the principle of convergent evolution is a thing. It's possible that and we cannot rule out that we are accidentally stumbling our way into homologous models of biological compute (I personally doubt it).
  - mjburgess 5y ago
    Well in the sense that the activation map over the brain is a function of space-time, ie., A(x, t) -- it is possible that the activations in a NN might instantiate this function... but that's literally probability 0. As the set of such functions, R^2 -> R are uncountable infinite.
    - dnautics 5y ago
      If you imagine the space of all possibilities as R^n -> R and the solution to be only one mapping function and not "some subspace of the possibility space", possibly around some ball of results, I got some news for you.

soraki_soladead 5y ago

Perceptrons were modeled after neurons in the sense of simple distributed computational units connected by weights. It doesn’t go any further than that. Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex. ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain). Backprop, however, takes no inspiration from biology.

dnautics 5y ago
> spatial receptive field. Yeap that's what I meant by inspired. It's a pretty weak inspiration. Thanks for the correction on ReLUs. Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.

mjburgess 5y ago

It was never an accurate model (in the sense that it never modelled, eg., plasticity and many more things). It was however an attempt to actually model activation patterns as boolean circuits. However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.) Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function. It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour. The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.

dnautics 5y ago
> its somewhat incoherent to suppose it does Agreed, but the principle of convergent evolution is a thing. It's possible that and we cannot rule out that we are accidentally stumbling our way into homologous models of biological compute (I personally doubt it).
- mjburgess 5y ago
  Well in the sense that the activation map over the brain is a function of space-time, ie., A(x, t) -- it is possible that the activations in a NN might instantiate this function... but that's literally probability 0. As the set of such functions, R^2 -> R are uncountable infinite.
  - dnautics 5y ago
    If you imagine the space of all possibilities as R^n -> R and the solution to be only one mapping function and not "some subspace of the possibility space", possibly around some ball of results, I got some news for you.