5 ms·

> The truth is that even the experts don’t completely understand how neural nets work. I'm no AI/ML expert, but I can't believe this is true... Is it?

hiddencost 11y ago
Extremely true.
oh_sigh 11y ago
Abstractly, we understand how neural nets work. However, looking at a specific trained neural net, it can be difficult to determine the exact reason why certain weights are they way they are, and what effect they have on the whole. Just like how we can know somewhat how neurons work from a modeling perspective, but when you bundle millions of them together, what exactly each one is doing is not quite clear.
- hiddencost 11y ago
  We understand how they work mechanically, but not why they work from a theoretical stand-point (which is I assume what you're trying to say by "abstractly"). Why it is that ASGD and backprop converges on a non-convex optimization problem, and what kinds of model topologies make it do better / worse? That's all basically art right now.
  - Houshalter 11y ago
    No that's not what he's trying to say. We know why gradient descent and back-propagation works. But no one understands what an actual trained neural net is doing. You can look at the weights, and you can watch the inputs and outputs, but it is very difficult to understand why it does what it does. It just fit a model to data, but there is no explanation why that model is best. The weights are not interprettable by humans. There have been some attempts at making models which humans can interpret. One program named Eureqa fit the simplest possible mathematical expression possible to a set of samples. A biologist tried it on his data and found that it created an expression which actually fit the data really well. But he couldn't publish it because he couldn't explain why it worked. It just did. But there was no understanding, no explanation.
    - dhj 11y ago
      Do you have a citation or blog post for this application of Eureqa that gave a good result but wasn't publishable from a biological study perspective? I have a background in BME and CS and am very interested in learning about this anecdote.
      - Houshalter 11y ago
        It was in an interview on an episode of radiolab: http://www.radiolab.org/story/91712-limits-of-science/ http://www.radiolab.org/story/91712-limits-of-science/ There aren't many details. I believe the scientist who said it is Dr. Gurol Suel, from that radiolab page.
    - Someone 11y ago
      "We know why gradient descent and back-propagation works." I would phrase that differently: we know when gradient descent and back-propagation work, not why they work for so many real world problems. For the when, there are zillions of published mathematical results stating "if a problem has property X, this and this method will find a solution with property Y in time T", in zillions of variations (Y can be the true optimum, a value within x% of the real optimum, the true optimum 'most of the time', etc. T can be 'eventually', 'after O(n^3) iterations', 'always', etc) However, for most real-world problems we do not know whether they have property X, or even how to go about arguing that it is likely they have property X, other than the somewhat circular "algorithm A seems to work well for it, and we know it works for problems of type X"
      - Leszek 11y ago
        Well, we do know why gradient descent works (for smooth data), at least for finding a local minimum, because finding the minimum is basically what it does by construction. Similarly, we certainly know how back-propagation works, because it's simple calculus, backwards application of the chain rule. Perhaps what you're trying to say is we don't know why finding local minima of these problems is good at solving the problem?
    - argonaut 11y ago
johnm1019 11y ago
In my experience doing 3D image reconstruction from 1D and 2D sensors, there are often situations where the engineers can only explain what's going on using math equations - and not in any other qualitative kind of way. For some people, this might be considered not "completely understanding" something.
- niels_olson 11y ago
  I submit I know of no one who can honestly and verifiably say they understand quantum spin without equations either. It just falls out of the math that way and is highly verifiable by experiment.
  - goldfeld 11y ago
    Can't you at least explain the fundamental need for quantum spin (the why) out of symmetry principles of the universe?
- argonaut 11y ago
  But nothing in neural nets "falls out" of some underlying natural/physical equations. It is entirely human-constructed engineering.
dantillberg 11y ago
Neural networks are really just a fancy statistical regression. Just the same as you could run a high-degree polynomial regression on some data and "not really know" why it chose the coefficients it did or why it yields a particular output for another input, neural networks typically involve a lot of mysterious coefficients, and it's hard to qualitatively explain why the neural network gives a particular output for another input. But in both cases, the underlying principles and mechanism of action are very well understood.
- argonaut 11y ago
  That's not true. Depending on the method for polynomial regression, you do know why it chose the coefficients. Furthermore, you know exactly what your model looks like, because you're dictating what the model is explicitly! (... a polynomial). In contrast to a convolutional neural net where you might have no idea what the filters in the third (not first) convolutional layer are doing. This is also a pretty bad example because polynomial regression is not giving amazing groundbreaking results. Polynomial regression has been known for centuries and gives pretty meh results in the real world.
  - nl 11y ago
    As an aside to argonaut's response (which is correct) it is important to note that there is a lot of ongoing work in understanding what the hidden layers in a CNN are doing. For image-classification type problems, it's pretty easy to pass an image in to the network and output the features each layer extracts. Indeed, many recent systems rely on CNN-extracted features being passed to a SVN classifier to classify images that the CNN wasn't trained on (because the SVN is easier to train will less data, and using the CNN frees one from doing feature engineering).
    - argonaut 11y ago
      We have a good understanding of the first CNN layer (they turn out to be edge/blob detectors). But later layers are not well understood.
ericflo 11y ago
Determining why a particular neuron in a large model fired or did not fire is still a guessing game, as far as I know. Kind of fun to look at patterns of when certain neurons activate a few layers deep in image classification convnets https://youtu.be/ta5fdaqDT3M?t=27m42s https://youtu.be/ta5fdaqDT3M?t=27m42s
sushirain 11y ago
Deep learning experts understand how neural nets work. But: 1. They don't usually try to explain the specific activations in the hidden layers of the network. This is hard and depends on the specific trained net. 2. They can't guarantee a net's performance before experimenting with it.

hiddencost 11y ago

Extremely true.

oh_sigh 11y ago

Abstractly, we understand how neural nets work. However, looking at a specific trained neural net, it can be difficult to determine the exact reason why certain weights are they way they are, and what effect they have on the whole. Just like how we can know somewhat how neurons work from a modeling perspective, but when you bundle millions of them together, what exactly each one is doing is not quite clear.

hiddencost 11y ago
We understand how they work mechanically, but not why they work from a theoretical stand-point (which is I assume what you're trying to say by "abstractly"). Why it is that ASGD and backprop converges on a non-convex optimization problem, and what kinds of model topologies make it do better / worse? That's all basically art right now.
- Houshalter 11y ago
  No that's not what he's trying to say. We know why gradient descent and back-propagation works. But no one understands what an actual trained neural net is doing. You can look at the weights, and you can watch the inputs and outputs, but it is very difficult to understand why it does what it does. It just fit a model to data, but there is no explanation why that model is best. The weights are not interprettable by humans. There have been some attempts at making models which humans can interpret. One program named Eureqa fit the simplest possible mathematical expression possible to a set of samples. A biologist tried it on his data and found that it created an expression which actually fit the data really well. But he couldn't publish it because he couldn't explain why it worked. It just did. But there was no understanding, no explanation.
  - dhj 11y ago
    Do you have a citation or blog post for this application of Eureqa that gave a good result but wasn't publishable from a biological study perspective? I have a background in BME and CS and am very interested in learning about this anecdote.
    - Houshalter 11y ago
      It was in an interview on an episode of radiolab: http://www.radiolab.org/story/91712-limits-of-science/ http://www.radiolab.org/story/91712-limits-of-science/ There aren't many details. I believe the scientist who said it is Dr. Gurol Suel, from that radiolab page.
  - Someone 11y ago
    "We know why gradient descent and back-propagation works." I would phrase that differently: we know when gradient descent and back-propagation work, not why they work for so many real world problems. For the when, there are zillions of published mathematical results stating "if a problem has property X, this and this method will find a solution with property Y in time T", in zillions of variations (Y can be the true optimum, a value within x% of the real optimum, the true optimum 'most of the time', etc. T can be 'eventually', 'after O(n^3) iterations', 'always', etc) However, for most real-world problems we do not know whether they have property X, or even how to go about arguing that it is likely they have property X, other than the somewhat circular "algorithm A seems to work well for it, and we know it works for problems of type X"
    - Leszek 11y ago
      Well, we do know why gradient descent works (for smooth data), at least for finding a local minimum, because finding the minimum is basically what it does by construction. Similarly, we certainly know how back-propagation works, because it's simple calculus, backwards application of the chain rule. Perhaps what you're trying to say is we don't know why finding local minima of these problems is good at solving the problem?
  - argonaut 11y ago

johnm1019 11y ago

In my experience doing 3D image reconstruction from 1D and 2D sensors, there are often situations where the engineers can only explain what's going on using math equations - and not in any other qualitative kind of way. For some people, this might be considered not "completely understanding" something.

niels_olson 11y ago
I submit I know of no one who can honestly and verifiably say they understand quantum spin without equations either. It just falls out of the math that way and is highly verifiable by experiment.
- goldfeld 11y ago
  Can't you at least explain the fundamental need for quantum spin (the why) out of symmetry principles of the universe?
argonaut 11y ago
But nothing in neural nets "falls out" of some underlying natural/physical equations. It is entirely human-constructed engineering.

dantillberg 11y ago

Neural networks are really just a fancy statistical regression. Just the same as you could run a high-degree polynomial regression on some data and "not really know" why it chose the coefficients it did or why it yields a particular output for another input, neural networks typically involve a lot of mysterious coefficients, and it's hard to qualitatively explain why the neural network gives a particular output for another input. But in both cases, the underlying principles and mechanism of action are very well understood.

argonaut 11y ago
That's not true. Depending on the method for polynomial regression, you do know why it chose the coefficients. Furthermore, you know exactly what your model looks like, because you're dictating what the model is explicitly! (... a polynomial). In contrast to a convolutional neural net where you might have no idea what the filters in the third (not first) convolutional layer are doing. This is also a pretty bad example because polynomial regression is not giving amazing groundbreaking results. Polynomial regression has been known for centuries and gives pretty meh results in the real world.
- nl 11y ago
  As an aside to argonaut's response (which is correct) it is important to note that there is a lot of ongoing work in understanding what the hidden layers in a CNN are doing. For image-classification type problems, it's pretty easy to pass an image in to the network and output the features each layer extracts. Indeed, many recent systems rely on CNN-extracted features being passed to a SVN classifier to classify images that the CNN wasn't trained on (because the SVN is easier to train will less data, and using the CNN frees one from doing feature engineering).
  - argonaut 11y ago
    We have a good understanding of the first CNN layer (they turn out to be edge/blob detectors). But later layers are not well understood.

ericflo 11y ago

Determining why a particular neuron in a large model fired or did not fire is still a guessing game, as far as I know. Kind of fun to look at patterns of when certain neurons activate a few layers deep in image classification convnets https://youtu.be/ta5fdaqDT3M?t=27m42s https://youtu.be/ta5fdaqDT3M?t=27m42s

sushirain 11y ago

Deep learning experts understand how neural nets work. But: 1. They don't usually try to explain the specific activations in the hidden layers of the network. This is hard and depends on the specific trained net. 2. They can't guarantee a net's performance before experimenting with it.