5 ms·

You have to remember what came before 2012: SVMs, Random Forests etc, absolutely nothing like the brain (yes, NNs are old, but 2012 was the start of the deep le

You have to remember what came before 2012: SVMs, Random Forests etc, absolutely nothing like the brain (yes, NNs are old, but 2012 was the start of the deep learning revolution). With this frame of reference, the brain and neural networks are both a kind of Connectionism with similar properties, and I think it makes perfect sense to liken them with each other, draw inspiration from one and apply it to the other.

signa11 2y ago
sorry, but i think neural-networks came way before 2012, notably the works of rumelhart, mccleland etc. see the 2 volume "parallel distributed processing" to read almost all about it. the book(s): https://direct.mit.edu/books/monograph/4424/Parallel-Distributed-Processing-Volume https://direct.mit.edu/books/monograph/4424/Parallel-Distrib... a-talk: https://www.youtube.com/watch?v=yQbJNEhgYUw https://www.youtube.com/watch?v=yQbJNEhgYUw
- FL33TW00D 2y ago
  I raise you Warren McCulloch in 1962: https://www.youtube.com/watch?v=wawMjJUCMVw https://www.youtube.com/watch?v=wawMjJUCMVw
- mcshicks 2y ago
  Jets and Sharks! https://github.com/acmiceli/IACModel https://github.com/acmiceli/IACModel
- martindbp 2y ago
  I knew someone would bring it up, which is why I added "(yes, NNs are old, but 2012 was the start of the deep learning revolution)"
  - versteegen 2y ago
    2012 was when the revolutionaries stormed the bastille and overthrew the old guard. But I say it was 2006 when the revolution started, when the manifesto was published: deep NNs can be trained end-to-end, learning their own features [1]. I think this is when "Deep Learning" became a term of art, and the paper has 24k citations. (Interestingly in a talk a Vector Hinton gave two weeks ago he said his paper on deep learning at NIPS 2006 was rejected because they already had one.) [1] G. E. Hinton and R. R. Salakhutdinov, 2006, Science, Reducing the Dimensionality of Data with Neural Networks
zitterbewegung 2y ago
Neural Networks are 200 years old (Legendre and Gauss defined Feed forward neural networks). Deep learning. The real difference between traditional ones and deep learning is a hierarchy of layers (hidden layers) which do different things to accomplish a goal. Even the concept of training is to provide weights on the neural network and there are many algorithms to do refinement, optimization and the network design.
- varjag 2y ago
  Gauss did not define feed forward neural networks, it all stems from a tweet of a very confused person.
- mrbungie 2y ago
  I mean, sure, you can model a simple linear regression fitted via Least Squares (pretty much what they did 200 years ago) with a one hidden layer feed-fwd Neural Network, but the theorical framework for NNs is quite different.
  - hervature 2y ago
    For Least Squares, you do not even use a hidden layer. Just a single dense layer from input directly to output. You also do not use an activation function (or use the identity activation function). That is, everything that makes neural networks special.
belter 2y ago
It is also odd to see such a weak argument as the brain-to-body mass ratio being used, as here: https://youtu.be/YD-9NG1Ke5Y?t=593 https://youtu.be/YD-9NG1Ke5Y?t=593 If this metric were truly indicative, what should we make of the remarkable ratios found in small birds (1:12), tree shrews (1:10), or even small ants (1:7)? https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio
- theptip 2y ago
  > what should we make of the remarkable ratios found… We also can’t implement those creatures’ control systems in silicon, so they too are doing things we can learn from?
zk4x 2y ago
What came before was regression. Which is to this day no 1 method if we want something interpretable, especially if we know which functions our variables follow. And self attention is very similar to correlation matrix. In a way neural networks are just bunch of regression models stacked on top of each other with some normalization and nonlinearity between them. It's cool however how closely it resembles biology.
bflesch 2y ago
Sorry, but they just called it "neuron" to sound nicer.

signa11 2y ago

sorry, but i think neural-networks came way before 2012, notably the works of rumelhart, mccleland etc. see the 2 volume "parallel distributed processing" to read almost all about it. the book(s): https://direct.mit.edu/books/monograph/4424/Parallel-Distributed-Processing-Volume https://direct.mit.edu/books/monograph/4424/Parallel-Distrib... a-talk: https://www.youtube.com/watch?v=yQbJNEhgYUw https://www.youtube.com/watch?v=yQbJNEhgYUw

FL33TW00D 2y ago
I raise you Warren McCulloch in 1962: https://www.youtube.com/watch?v=wawMjJUCMVw https://www.youtube.com/watch?v=wawMjJUCMVw
mcshicks 2y ago
Jets and Sharks! https://github.com/acmiceli/IACModel https://github.com/acmiceli/IACModel
martindbp 2y ago
I knew someone would bring it up, which is why I added "(yes, NNs are old, but 2012 was the start of the deep learning revolution)"
- versteegen 2y ago
  2012 was when the revolutionaries stormed the bastille and overthrew the old guard. But I say it was 2006 when the revolution started, when the manifesto was published: deep NNs can be trained end-to-end, learning their own features [1]. I think this is when "Deep Learning" became a term of art, and the paper has 24k citations. (Interestingly in a talk a Vector Hinton gave two weeks ago he said his paper on deep learning at NIPS 2006 was rejected because they already had one.) [1] G. E. Hinton and R. R. Salakhutdinov, 2006, Science, Reducing the Dimensionality of Data with Neural Networks

zitterbewegung 2y ago

Neural Networks are 200 years old (Legendre and Gauss defined Feed forward neural networks). Deep learning. The real difference between traditional ones and deep learning is a hierarchy of layers (hidden layers) which do different things to accomplish a goal. Even the concept of training is to provide weights on the neural network and there are many algorithms to do refinement, optimization and the network design.

varjag 2y ago
Gauss did not define feed forward neural networks, it all stems from a tweet of a very confused person.
mrbungie 2y ago
I mean, sure, you can model a simple linear regression fitted via Least Squares (pretty much what they did 200 years ago) with a one hidden layer feed-fwd Neural Network, but the theorical framework for NNs is quite different.
- hervature 2y ago
  For Least Squares, you do not even use a hidden layer. Just a single dense layer from input directly to output. You also do not use an activation function (or use the identity activation function). That is, everything that makes neural networks special.

belter 2y ago

It is also odd to see such a weak argument as the brain-to-body mass ratio being used, as here: https://youtu.be/YD-9NG1Ke5Y?t=593 https://youtu.be/YD-9NG1Ke5Y?t=593 If this metric were truly indicative, what should we make of the remarkable ratios found in small birds (1:12), tree shrews (1:10), or even small ants (1:7)? https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio

theptip 2y ago
> what should we make of the remarkable ratios found… We also can’t implement those creatures’ control systems in silicon, so they too are doing things we can learn from?

zk4x 2y ago

What came before was regression. Which is to this day no 1 method if we want something interpretable, especially if we know which functions our variables follow. And self attention is very similar to correlation matrix. In a way neural networks are just bunch of regression models stacked on top of each other with some normalization and nonlinearity between them. It's cool however how closely it resembles biology.

bflesch 2y ago

Sorry, but they just called it "neuron" to sound nicer.