4 ms·

How good are we at explaining large ML model outputs? My understanding is not very good? And that is with us having a very deep understanding of the hardware,

How good are we at explaining large ML model outputs? My understanding is not very good? And that is with us having a very deep understanding of the hardware, software, algorithms, and mathematics involved. Now let’s suppose that you run your ML model on a computer and you get bad results. Two experts show up. One says that you have a faulty memory chip and that they can fix what is wrong with the memory chip and then the model should work. The other says that because of the training data the model was exposed to, some of the weights are bad, and by providing focused new data for the model to train with, they can adjust those weights so the model produces better output. While you are deciding, these two then get into a big fight each accusing the other of being a charlatan.

matheusmoreira 4y ago
And neither of them really knows anything, they just came up with reasonable hypotheses to explain observed behavior, figured out what to do about them, ran tests to confirm and then approved those treatments on the basis that they worked more often than not. What does this stuff even do? Not sure, they just have proof that it's effective.
deleted 4y ago
[deleted]
- deleted 4y ago
  [deleted]
mjburgess 4y ago
I'm unsure as to the thesis of this comment: what's your point? Perhaps I just don't buy your premise. We understand ML models very-well, but we don't like the explanation. Models are on a spectrum from parametric where a data distribution is assumed, to non-parametric, where it isnt. In the non-param case, models largely remember compressed "averages" of historical data. The sense in which we dont "understand" how, eg., a NN comes to a prediction is only in the sense that we cannot, a priori, locate the specific subset of historical data that it has taken a weighted average-of... but we know well that this is what its doing. ie., something incredibly dumb and boring.
- avgcorrection 4y ago
  > I'm unsure as to the thesis of this comment: what's your point? A roundabout analogy which is supposed to liken the human mind to ML by showing that experts in both fields are equally perplexed at the two objects of study (ML and the human mind). You see now how we are approaching human-like intelligence? ML experts don’t know what they are making or doing, just like psychologists and psychiatrists can’t communicate on their different models of the mind!
  - haswell 4y ago
    > just like psychologists and psychiatrists can’t communicate on their different models of the mind! This is where the analogy breaks down quickly, though. I realize that the article tells a story about extreme situations where the two were directly at odds, but that is not the state of the art in psychology/psychiatry. Psychologists and psychiatrists routinely communicate with each other about their models of the mind, and collaborate on the appropriate course of treatment for patients. For many of the same reasons that some injuries require surgery to fix a serious issue, medication to deal with short term pain, and physical therapy to rebuild the body.

matheusmoreira 4y ago

And neither of them really knows anything, they just came up with reasonable hypotheses to explain observed behavior, figured out what to do about them, ran tests to confirm and then approved those treatments on the basis that they worked more often than not. What does this stuff even do? Not sure, they just have proof that it's effective.

deleted 4y ago

[deleted]

deleted 4y ago
[deleted]

mjburgess 4y ago

I'm unsure as to the thesis of this comment: what's your point? Perhaps I just don't buy your premise. We understand ML models very-well, but we don't like the explanation. Models are on a spectrum from parametric where a data distribution is assumed, to non-parametric, where it isnt. In the non-param case, models largely remember compressed "averages" of historical data. The sense in which we dont "understand" how, eg., a NN comes to a prediction is only in the sense that we cannot, a priori, locate the specific subset of historical data that it has taken a weighted average-of... but we know well that this is what its doing. ie., something incredibly dumb and boring.

avgcorrection 4y ago
> I'm unsure as to the thesis of this comment: what's your point? A roundabout analogy which is supposed to liken the human mind to ML by showing that experts in both fields are equally perplexed at the two objects of study (ML and the human mind). You see now how we are approaching human-like intelligence? ML experts don’t know what they are making or doing, just like psychologists and psychiatrists can’t communicate on their different models of the mind!
- haswell 4y ago
  > just like psychologists and psychiatrists can’t communicate on their different models of the mind! This is where the analogy breaks down quickly, though. I realize that the article tells a story about extreme situations where the two were directly at odds, but that is not the state of the art in psychology/psychiatry. Psychologists and psychiatrists routinely communicate with each other about their models of the mind, and collaborate on the appropriate course of treatment for patients. For many of the same reasons that some injuries require surgery to fix a serious issue, medication to deal with short term pain, and physical therapy to rebuild the body.