5 ms·

requires all algorithms to "explain" their reasoning It's about the application. If you are detecting cat images on a social network, then a few false positive

by gaius 10y ago

requires all algorithms to "explain" their reasoning It's about the application. If you are detecting cat images on a social network, then a few false positives or negatives are totally no big deal. If your algo is making investment decisions or is being used as evidence in court then it is entirely reasonable to expect it to have a rock solid causal chain easily comprehensible by a layperson. The example in the article is noise reduction and the method was "we tweaked it until we got the results we wanted". In many fields that's called "overfitting" (or "fitting up") and is a big red flag.

backpropaganda 10y ago
All machine learning models are capable of overfitting. Deep learning does not have a special relationship to overfitting, and in fact, there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1]. Citing certain examples of misuse of deep learning by folks who "tweaked it till it worked" doesn't say anything about deep learning at large. The same team would do the same with a much less powerful model. Of course, being able to explain is a very valuable trait. But there's just no evidence that deep learning methods are any less amenable to interpretation than SVMs or decision trees. The latter models' "interpretation" is mostly stuff like "feature 543 and 632 were on" , while deep learning methods can not only do that, but can also synthesize examples of characteristics that the model looked for [2]. [1]: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530 [2]: https://arxiv.org/abs/1605.09304 https://arxiv.org/abs/1605.09304
- yorwba 10y ago
  Then requiring algorithms to explain their reasoning should give neural networks an edge over other methods, if they are really easier to interpret.
- gaius 10y ago
  Deep learning does not have a special relationship to overfitting Yes and no. Let's say you have a time series and you know there is some pattern in it and you are interested in predicting what it will be at time t. A Fourier analysis would give you a formula to plug numbers into that anyone can understand. NN or DL might give you as good, or accounting for noise, even a slightly better prediction when backtested but now you need to explain "why did it do this" - to a financial regulator who is looking at insider trades, or a commission looking into why a piece of equipment failed killing the user (say, a self-driving car), or a newspaper who's noticed more people of race X are being turned down for mortgages. That's why I say it is dependent on the application of the algo.
- YeGoblynQueenne 10y ago
  >> [SVM and Decision Trees'] "interpretation" is mostly stuff like "feature 543 and 632 were on" SVMs are well-known black-box optimisers of numerical parameters so not more or less interpretable than neural nets, but decision tree learners are a different matter. The model they build is symbolic- a propositional logic theorem that can be output in a format that is directly inspectable and interpretable by a human being with an understanding of propositional logic. It's true that once decision trees grow beyond a certain size they're hard to interpret, but even so they can make a lot more sense than a heap of numbers in a large graph. Neural nets used for image processing are special in that they can output their activations as images that are directly interpretable by humans, but that's about the only application where we can really tell what a deep network is learning. Edit: >> All machine learning models are capable of overfitting. Aye, but deep nets by definition build extremely complex models with millions of parameters and so are especially prone to overfitting. Bias-vs-variance and all that.
Asooka 10y ago
But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct? Edit: I should add that I worked on a similar problem. I can't reveal all the details, but it was about scoring pairs of objects to say how much one object is like the other. At first we used an off-the-shelf system that only spit out abstract values, so you can say "A is like B with a value of 5, but it's like C with a value of 5.17, so A is more like C than like B". The problem was those scores didn't correspond to any explainable metrics, so we switched to a custom system that could spit out a normalized value (0 to 1) and tell you exactly which qualities were alike and how the score was formed. People were NEVER happy with the values. Never. Everyone "felt" that these two should be more alike, or there should always be at least one object that has at least 0.8 likeness to another, etc. Plus, the values were now non-transitive, i.e. A can be 0.9 like B, but B is 0.2 like A. After about a year of tweaking we switched back to a non-explainable system. Of course, the "non-explainable" system is perfectly explainable, it's a computer program and it isn't even deep learning, it's just long equations, the explanation is just "this long equation results in this value". The semantics of that equation though are very hard to impossible to understand. Plus, I don't think we can have a semantic explanation of a deep learning system precisely because we use deep learning when we can't semantically specify an algorithmic solution to the problem.
- gaius 10y ago
  But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct? It depends who you are explaining it to, of course, different audiences will have different standards, and will have or have access to, different kinds of experts to validate the explanation. If the question is "why did your self-driving car hit that pedestrian?" for example, the standard would be pretty high. "It works most of the time but we don't know why it failed this time and we don't know when it might fail again" isn't going to fly. If the question is "we've just busted this trader for insider dealing, and you made the same trades at the same time, why?" then "the computer just told me to for no particular reason" isn't going to go down well either. You would need to show how data that you legitimately had access to reproducibly leads to exactly the same decision being made. What do you do if you don't like the explanation? That's easy. Someone's going to jail. Don't let that be you :-)
xherberta 10y ago
As Cathy O'Neil points out, black-box algos are deciding things that have a huge impact on individuals' lives: which teachers get fired, which prisoners don't get parole, and which loan applicants get approved and denied. In the latter two cases, the algos are tougher on minorities. It's troubling when we don't know why an algo is racist. We need ways of checking that these influential algos aren't reinforcing trends we would rather diminish.
- gaius 10y ago
  No-fly lists as well. reinforcing trends we would rather diminish Well, of course they are, at the end of the day all any predictive algo is doing is extrapolating a trend. It usually requires serious regulatory intervention - such as men being charged more for car insurance because "the computers said so" until legislation was passed barring sex as an input to the model.
  - sdenton4 10y ago
    The black box nature of the regression can make things difficult, though. If you have a couple other parameters which indicate 'black male' with high probability, you might find layer one reconstructing the notion of gender and layer two making decisions based on it.
    - moultano 10y ago
      Much better to include the feature in the training, and exclude it from the inference. That actually cancels out the effect, rather than just incentivizing the model to reconstruct it.
- Houshalter 10y ago
  No one cares about human judges that decide parole or to fire teachers. Humans are vastly worse. Attractive people are much more likely to get hired or get shorter prison sentences. Judges tend to give much harsher sentences just before lunch when they are hungry. Their predictions about what people are likely to reoffend are often worse than chance. Additionally, the study about the "racist algorithms" was fraudulent. Their results were "almost statistically significant". I.e. not statistically significant. Compared to human judges which have well studied biases by race. There's nothing remotely fair about human judgement and you should always prefer an algorithm. In almost every domain they could find, researchers have found that even simple linear regression beats predictions of human experts. And this has huge effects on our society. Humans making biased hiring decisions leads to mass discrimination against certain groups and very suboptimal employees. Having humans make loan decisions, means much higher interest rates, more people go bankrupt, and the economy grows much slower. The EU banning the use of algorithms is just absurd.
tensor 10y ago
For your court example, if you have human oversight it doesn't matter. For example, machine finds evidence, human looks at it the found evidence, verifies and explains it, then no problem. Also, I'd like to point out that much "evidence" in court is very far from rock solid. I'm always curious why we hold machines to higher standards than humans.

by gaius 10y ago

backpropaganda 10y ago

All machine learning models are capable of overfitting. Deep learning does not have a special relationship to overfitting, and in fact, there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1]. Citing certain examples of misuse of deep learning by folks who "tweaked it till it worked" doesn't say anything about deep learning at large. The same team would do the same with a much less powerful model. Of course, being able to explain is a very valuable trait. But there's just no evidence that deep learning methods are any less amenable to interpretation than SVMs or decision trees. The latter models' "interpretation" is mostly stuff like "feature 543 and 632 were on" , while deep learning methods can not only do that, but can also synthesize examples of characteristics that the model looked for [2]. [1]: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530 [2]: https://arxiv.org/abs/1605.09304 https://arxiv.org/abs/1605.09304

yorwba 10y ago
Then requiring algorithms to explain their reasoning should give neural networks an edge over other methods, if they are really easier to interpret.
gaius 10y ago
Deep learning does not have a special relationship to overfitting Yes and no. Let's say you have a time series and you know there is some pattern in it and you are interested in predicting what it will be at time t. A Fourier analysis would give you a formula to plug numbers into that anyone can understand. NN or DL might give you as good, or accounting for noise, even a slightly better prediction when backtested but now you need to explain "why did it do this" - to a financial regulator who is looking at insider trades, or a commission looking into why a piece of equipment failed killing the user (say, a self-driving car), or a newspaper who's noticed more people of race X are being turned down for mortgages. That's why I say it is dependent on the application of the algo.
YeGoblynQueenne 10y ago
>> [SVM and Decision Trees'] "interpretation" is mostly stuff like "feature 543 and 632 were on" SVMs are well-known black-box optimisers of numerical parameters so not more or less interpretable than neural nets, but decision tree learners are a different matter. The model they build is symbolic- a propositional logic theorem that can be output in a format that is directly inspectable and interpretable by a human being with an understanding of propositional logic. It's true that once decision trees grow beyond a certain size they're hard to interpret, but even so they can make a lot more sense than a heap of numbers in a large graph. Neural nets used for image processing are special in that they can output their activations as images that are directly interpretable by humans, but that's about the only application where we can really tell what a deep network is learning. Edit: >> All machine learning models are capable of overfitting. Aye, but deep nets by definition build extremely complex models with millions of parameters and so are especially prone to overfitting. Bias-vs-variance and all that.

Asooka 10y ago

But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct? Edit: I should add that I worked on a similar problem. I can't reveal all the details, but it was about scoring pairs of objects to say how much one object is like the other. At first we used an off-the-shelf system that only spit out abstract values, so you can say "A is like B with a value of 5, but it's like C with a value of 5.17, so A is more like C than like B". The problem was those scores didn't correspond to any explainable metrics, so we switched to a custom system that could spit out a normalized value (0 to 1) and tell you exactly which qualities were alike and how the score was formed. People were NEVER happy with the values. Never. Everyone "felt" that these two should be more alike, or there should always be at least one object that has at least 0.8 likeness to another, etc. Plus, the values were now non-transitive, i.e. A can be 0.9 like B, but B is 0.2 like A. After about a year of tweaking we switched back to a non-explainable system. Of course, the "non-explainable" system is perfectly explainable, it's a computer program and it isn't even deep learning, it's just long equations, the explanation is just "this long equation results in this value". The semantics of that equation though are very hard to impossible to understand. Plus, I don't think we can have a semantic explanation of a deep learning system precisely because we use deep learning when we can't semantically specify an algorithmic solution to the problem.

gaius 10y ago
But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct? It depends who you are explaining it to, of course, different audiences will have different standards, and will have or have access to, different kinds of experts to validate the explanation. If the question is "why did your self-driving car hit that pedestrian?" for example, the standard would be pretty high. "It works most of the time but we don't know why it failed this time and we don't know when it might fail again" isn't going to fly. If the question is "we've just busted this trader for insider dealing, and you made the same trades at the same time, why?" then "the computer just told me to for no particular reason" isn't going to go down well either. You would need to show how data that you legitimately had access to reproducibly leads to exactly the same decision being made. What do you do if you don't like the explanation? That's easy. Someone's going to jail. Don't let that be you :-)

xherberta 10y ago

As Cathy O'Neil points out, black-box algos are deciding things that have a huge impact on individuals' lives: which teachers get fired, which prisoners don't get parole, and which loan applicants get approved and denied. In the latter two cases, the algos are tougher on minorities. It's troubling when we don't know why an algo is racist. We need ways of checking that these influential algos aren't reinforcing trends we would rather diminish.

gaius 10y ago
No-fly lists as well. reinforcing trends we would rather diminish Well, of course they are, at the end of the day all any predictive algo is doing is extrapolating a trend. It usually requires serious regulatory intervention - such as men being charged more for car insurance because "the computers said so" until legislation was passed barring sex as an input to the model.
- sdenton4 10y ago
  The black box nature of the regression can make things difficult, though. If you have a couple other parameters which indicate 'black male' with high probability, you might find layer one reconstructing the notion of gender and layer two making decisions based on it.
  - moultano 10y ago
    Much better to include the feature in the training, and exclude it from the inference. That actually cancels out the effect, rather than just incentivizing the model to reconstruct it.
Houshalter 10y ago
No one cares about human judges that decide parole or to fire teachers. Humans are vastly worse. Attractive people are much more likely to get hired or get shorter prison sentences. Judges tend to give much harsher sentences just before lunch when they are hungry. Their predictions about what people are likely to reoffend are often worse than chance. Additionally, the study about the "racist algorithms" was fraudulent. Their results were "almost statistically significant". I.e. not statistically significant. Compared to human judges which have well studied biases by race. There's nothing remotely fair about human judgement and you should always prefer an algorithm. In almost every domain they could find, researchers have found that even simple linear regression beats predictions of human experts. And this has huge effects on our society. Humans making biased hiring decisions leads to mass discrimination against certain groups and very suboptimal employees. Having humans make loan decisions, means much higher interest rates, more people go bankrupt, and the economy grows much slower. The EU banning the use of algorithms is just absurd.

tensor 10y ago

For your court example, if you have human oversight it doesn't matter. For example, machine finds evidence, human looks at it the found evidence, verifies and explains it, then no problem. Also, I'd like to point out that much "evidence" in court is very far from rock solid. I'm always curious why we hold machines to higher standards than humans.