4 ms·

I was told ~30 years ago by a leading computer scientist in the NN field that biology has nothing to teach us in terms of implementation. I switched from CS to

I was told ~30 years ago by a leading computer scientist in the NN field that biology has nothing to teach us in terms of implementation. I switched from CS to neuroscience anyway. I've wrestled with his statement ever since. I'll say that nothing I've seen since then has shown him wrong.

ericjang 6y ago
OP here. I hope that's not the takeaway readers glean from my article - the point I was making was just that it doesn't make sense to shoehorn a biophysical learning mechanism into a DNN, rather we should use a DNN to find a biophysical learning mechanism. As to whether biophysical learning has anything to teach us is an entirely different question which I don't discuss in the post.
- intrasight 6y ago
  I did get and agree with your point. And I do see great benefits using DNN to study biophysical learning mechanisms. I was indeed addressing that different question, but it's also an interesting one.
xkcd-sucks 6y ago
Nobody understands the biology fully enough to drive a better ML implementation. Individual neurons are very complex and their interactions even more so
dragontamer 6y ago
You understand airplanes thanks to air foils, not the study of birds, bumblebees, or bats. Not that it's wasteful to study birds. It's just somewhat ironic that studying birds teaches us very little about practical flight. --------- I think it's important to learn the human mechanism for learning. It's clearly not backprop, any more than a propeller compares to bird wings. Understanding human learning would have huge implications for the field of education, rhetoric, marketing, UI design and more.
- akdor1154 6y ago
  The idea of adtech knowing how the physical process of learning works is terrifying.
  - titzer 6y ago
    What is the physical process of learning? There's just one now? Adtech will never stop; it will spawn baby universes replete with simulated human beings complete with bad breath and failing hair follicles if they're convinced they can make one more ad dollar.
jillesvangurp 6y ago
My neural networks teacher in university joked that "if you come here to study brains you are in the wrong place because this is a linear algebra course". I think that's a fair characterization and I wasn't very good at it/interested in it (the math part) and subsequently dropped out. However, it's a field that clearly takes its inspiration from the way biological brains work and then reduces that to a mathematical problem that we know how to optimize. It's a bit like not seeing the trees for the forest. There's a lot of matrix computation going on but in the end it's maybe not about that. My observation of the field 25 years later is that it's still dominated by a lot of people good at math that are at this point generating results that are definitely very interesting yet not fully understood. We've gotten very good at wiring together matrix computations in interesting ways and training them to do useful stuff. But just like with real brains, we struggle to understand how the resulting networks actually work or how to design them properly. I've seen high school kids work with tensor flow and they can definitely pull some magic tricks with that but they'd probably struggle to explain how the magic works as it's basically a black box component for them. And I suspect that is the case for large parts of the machine learning community as well. Math makes all of the lower level stuff work obviously but it's not a great tool for the more complex behavior that is basically emergent complexity of these networks. We can put them together and tinker with them but we seem to not have a deep understanding of how they work or even why they work. Very much like real brains.
kaba0 6y ago
Is there some truth to the “reverse” though —- that is, is the emerging patterns similar for similarish problems? What comes to mind is the similar first-layers in the human vision and google’s vision AI, with vertical/horizontal lines being “matched”.

ericjang 6y ago

OP here. I hope that's not the takeaway readers glean from my article - the point I was making was just that it doesn't make sense to shoehorn a biophysical learning mechanism into a DNN, rather we should use a DNN to find a biophysical learning mechanism. As to whether biophysical learning has anything to teach us is an entirely different question which I don't discuss in the post.

intrasight 6y ago
I did get and agree with your point. And I do see great benefits using DNN to study biophysical learning mechanisms. I was indeed addressing that different question, but it's also an interesting one.

xkcd-sucks 6y ago

Nobody understands the biology fully enough to drive a better ML implementation. Individual neurons are very complex and their interactions even more so

dragontamer 6y ago

You understand airplanes thanks to air foils, not the study of birds, bumblebees, or bats. Not that it's wasteful to study birds. It's just somewhat ironic that studying birds teaches us very little about practical flight. --------- I think it's important to learn the human mechanism for learning. It's clearly not backprop, any more than a propeller compares to bird wings. Understanding human learning would have huge implications for the field of education, rhetoric, marketing, UI design and more.

akdor1154 6y ago
The idea of adtech knowing how the physical process of learning works is terrifying.
- titzer 6y ago
  What is the physical process of learning? There's just one now? Adtech will never stop; it will spawn baby universes replete with simulated human beings complete with bad breath and failing hair follicles if they're convinced they can make one more ad dollar.

jillesvangurp 6y ago

My neural networks teacher in university joked that "if you come here to study brains you are in the wrong place because this is a linear algebra course". I think that's a fair characterization and I wasn't very good at it/interested in it (the math part) and subsequently dropped out. However, it's a field that clearly takes its inspiration from the way biological brains work and then reduces that to a mathematical problem that we know how to optimize. It's a bit like not seeing the trees for the forest. There's a lot of matrix computation going on but in the end it's maybe not about that. My observation of the field 25 years later is that it's still dominated by a lot of people good at math that are at this point generating results that are definitely very interesting yet not fully understood. We've gotten very good at wiring together matrix computations in interesting ways and training them to do useful stuff. But just like with real brains, we struggle to understand how the resulting networks actually work or how to design them properly. I've seen high school kids work with tensor flow and they can definitely pull some magic tricks with that but they'd probably struggle to explain how the magic works as it's basically a black box component for them. And I suspect that is the case for large parts of the machine learning community as well. Math makes all of the lower level stuff work obviously but it's not a great tool for the more complex behavior that is basically emergent complexity of these networks. We can put them together and tinker with them but we seem to not have a deep understanding of how they work or even why they work. Very much like real brains.

kaba0 6y ago

Is there some truth to the “reverse” though —- that is, is the emerging patterns similar for similarish problems? What comes to mind is the similar first-layers in the human vision and google’s vision AI, with vertical/horizontal lines being “matched”.