3 ms·

As someone with a strong background in Biology who took several AI classes at an Ivy League school, I found many of my CS professors models oversimplified neuro

by slacka 13y ago

As someone with a strong background in Biology who took several AI classes at an Ivy League school, I found many of my CS professors models oversimplified neurons to absurd levels. Teaching CS students to use perceptron that can't even change its own behavior is like teaching MechEs to built rockets out of Legos. Sure you can do some cool things with them, but the sooner you move on, the sooner you can make some real progress. After 2 AI winters, I'm happy to see that many schools like Berkeley are finally starting to combine their neuroscience and AI labs.

jergosh 13y ago
Neurons in AI are not supposed to exactly mimic the behaviour of 'real' neurons...
jerf 13y ago
jergosh is right too, but I think the real reason is that academic computer scientists want models they can mathematically reason about. Real neurons are fine and dandy, and can obviously do fantastic things like drive muscles in certain ways to post semantically meaningful content to Hacker News, but we still don't have a general model that can explain how they work in concert at scale. Perceptron's popularity arises not from their accuracy... I'm pretty sure everybody knew they were inaccurate models, though the hope was (falsely) that the inaccuracies wouldn't matter. Their popularity derives from the discovery of the backpropagation algorithm. In fact, if one were to take a mathematical view of the situation, arguably backpropagation is what was really discovered, and perceptrons are incidentally the moving parts in the backpropagation algorithm. It is not generally useful in AI to have some horrifically complicated algorithm that is based on something vaguely realistic, but that nobody know how to make do anything, or update based on new input, or figure out what's wrong when it doesn't work.
- slacka 13y ago
  I have no issue with using perceptrons as a learning tool. But they have clung to this model for far too long. The main problem I see is the biogical learning mechanism comes from some feedback mechanism that causes the neurons to grown new connections. Perceptrons have no way to mimic this and instead resort to sending an error from the output units back toward the input units duringing the learning phase. But when we go to use them, the information only flows forward. So you can never produce a dynamic learning system with these Legos. No matter how much super-glue you use, this rockets never going to make it into space. Yet many of these new deep learning neural nets continue to use back propagation for their learning phase. I never said we needed EXACT models. And, I know perceptrons are simple to model mathematically and have 40 years of research and tools backing them. But after 40 years, isn't it time we try just a little harder to make something that is a little closer to how biological neurons really work?
  - jerf 13y ago
    "But after 40 years, isn't it time we try just a little harder to make something that is a little closer to how biological neurons really work?" We have been trying. It has proved to be very difficult to get them to do anything useful, though, which was my point. Dedicated AI researchers are actually intensely aware of the differences between real neurons and their neurons, but getting fake neurons to do anything useful has not been as easy as just "simulate what the neurons do". There was an interesting breakthrough 6 months to a year ago, some new algorithm that is apparently able to update a more interesting neural net, but I don't know much more than that at the moment.
    - slacka 13y ago
      I agree. From what I've read many of the younger AI researchers are interested. As I mentioned earlier, some school have recently created special labs and programs for them to work together. My point was just 10 years ago, none of my CS professors showed much interest in neurobiology. Fortunately I think this attitude is changing.

by slacka 13y ago

jergosh 13y ago

Neurons in AI are not supposed to exactly mimic the behaviour of 'real' neurons...

jerf 13y ago

jergosh is right too, but I think the real reason is that academic computer scientists want models they can mathematically reason about. Real neurons are fine and dandy, and can obviously do fantastic things like drive muscles in certain ways to post semantically meaningful content to Hacker News, but we still don't have a general model that can explain how they work in concert at scale. Perceptron's popularity arises not from their accuracy... I'm pretty sure everybody knew they were inaccurate models, though the hope was (falsely) that the inaccuracies wouldn't matter. Their popularity derives from the discovery of the backpropagation algorithm. In fact, if one were to take a mathematical view of the situation, arguably backpropagation is what was really discovered, and perceptrons are incidentally the moving parts in the backpropagation algorithm. It is not generally useful in AI to have some horrifically complicated algorithm that is based on something vaguely realistic, but that nobody know how to make do anything, or update based on new input, or figure out what's wrong when it doesn't work.

slacka 13y ago
I have no issue with using perceptrons as a learning tool. But they have clung to this model for far too long. The main problem I see is the biogical learning mechanism comes from some feedback mechanism that causes the neurons to grown new connections. Perceptrons have no way to mimic this and instead resort to sending an error from the output units back toward the input units duringing the learning phase. But when we go to use them, the information only flows forward. So you can never produce a dynamic learning system with these Legos. No matter how much super-glue you use, this rockets never going to make it into space. Yet many of these new deep learning neural nets continue to use back propagation for their learning phase. I never said we needed EXACT models. And, I know perceptrons are simple to model mathematically and have 40 years of research and tools backing them. But after 40 years, isn't it time we try just a little harder to make something that is a little closer to how biological neurons really work?
- jerf 13y ago
  "But after 40 years, isn't it time we try just a little harder to make something that is a little closer to how biological neurons really work?" We have been trying. It has proved to be very difficult to get them to do anything useful, though, which was my point. Dedicated AI researchers are actually intensely aware of the differences between real neurons and their neurons, but getting fake neurons to do anything useful has not been as easy as just "simulate what the neurons do". There was an interesting breakthrough 6 months to a year ago, some new algorithm that is apparently able to update a more interesting neural net, but I don't know much more than that at the moment.
  - slacka 13y ago
    I agree. From what I've read many of the younger AI researchers are interested. As I mentioned earlier, some school have recently created special labs and programs for them to work together. My point was just 10 years ago, none of my CS professors showed much interest in neurobiology. Fortunately I think this attitude is changing.