3 ms·

You cannot then use such a network to make a questionable decision. In other words, they better be robust or you might drown in lawsuits or face a big fat one.

You cannot then use such a network to make a questionable decision. In other words, they better be robust or you might drown in lawsuits or face a big fat one. Which is the exact result people want. We want explanations that are valid and understandable in relatively plain language, using postulate logic.

tzs 8y ago
It's not actually hard to come up with a valid and understandable explanation from a neural net, or from any other black box decision maker as long as the decision maker is consistent. Let's take simplified case, to make visualization easier. Suppose the black box is deciding whether to approve mortgages, and it only has three inputs: income, credit score, and mortgage amount. You can visualize the inputs as points in a 3-dimensional space, each point representing a unique combination of income, credit score, and mortgage amount. Now imagine in that space we place green dots at all the points that represent inputs that result in approval of the mortgage application. Do nothing at the points for inputs that represent rejected inputs. Need to give someone an explanation of their rejection? Find their point in that space, and then find points on the green region that are nearest their point, and tell them what would need to change to move their point to those green points. I'd suggest, in this 3-dimensional space, to look at four green region points. (1) the nearest green point reached moving from their point parallel to the income axis, (2) same but using credit score axis, (3) same but using mortgage amount axis, and (4) the nearest green point.
kirkules 8y ago
In the context of a decision not made by an algorithm, would you find acceptable an explanation like "many repeated studies from well-known and reputed institutions and universities have concluded that X leads to Y, and therefore as a way of providing Y to our customers, we have chosen to do X"? In case your answer is yes, would it be acceptable to use the track record of a particular network and similarly trained networks, and why or why not?

tzs 8y ago

It's not actually hard to come up with a valid and understandable explanation from a neural net, or from any other black box decision maker as long as the decision maker is consistent. Let's take simplified case, to make visualization easier. Suppose the black box is deciding whether to approve mortgages, and it only has three inputs: income, credit score, and mortgage amount. You can visualize the inputs as points in a 3-dimensional space, each point representing a unique combination of income, credit score, and mortgage amount. Now imagine in that space we place green dots at all the points that represent inputs that result in approval of the mortgage application. Do nothing at the points for inputs that represent rejected inputs. Need to give someone an explanation of their rejection? Find their point in that space, and then find points on the green region that are nearest their point, and tell them what would need to change to move their point to those green points. I'd suggest, in this 3-dimensional space, to look at four green region points. (1) the nearest green point reached moving from their point parallel to the income axis, (2) same but using credit score axis, (3) same but using mortgage amount axis, and (4) the nearest green point.

kirkules 8y ago

In the context of a decision not made by an algorithm, would you find acceptable an explanation like "many repeated studies from well-known and reputed institutions and universities have concluded that X leads to Y, and therefore as a way of providing Y to our customers, we have chosen to do X"? In case your answer is yes, would it be acceptable to use the track record of a particular network and similarly trained networks, and why or why not?