4 ms·

Explainability is the new credulity. The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underl

Explainability is the new credulity. The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted -- without needing to check the actual explanations. The NIST proposals are interesting, though, and it would be a good challenge to make them work. EDIT: adding commentary. The four principles (quoting Hullman quoting NIST) are: 1) AI systems should deliver accompanying evidence or reasons for all their outputs. 2) Systems should provide explanations that are meaningful or understandable to individual users. 3) The explanation correctly reflects the system’s process for generating the output. 4) The system only operates under conditions for which it was designed or when the system reaches a sufficient confidence in its output. (The idea is that if a system has insufficient confidence in its decision, it should not supply a decision to the user. What do I think Hullman is getting at with "the New Uncertainty"? A major theme in Hullman's research, reductively, is understanding and then helping how people make sense of uncertainty as mediated through data visualization. In an explanation system, replace a the viz with a narrative of a solution process, as carried out by some algo agent, or with some rationale or justification, by as carried out by same. The process of judgment is now uncertain as well. Explanations are uncertain narratives. How does a person make sense of that? The NIST #2 bar for what is "meaningful or understandable", without relying explicitly in the authority of the recommender, seems pretty high to me.

aeternum 5y ago
There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. I wonder if we could tell whether or not a sufficiently complex NN or AI were doing the same, or if it even matters. It at least feels somewhat useless if system A makes the decision but then system B comes up with justification (even if the justification is plausible).
- troelsSteegin 5y ago
  XAI research talks about exactly that, system B "surrogate models" that are consistent with the recommender right around the locus of a prediction. If the surrogate model is easy to interpret, you can forgo directly explaining the base recommender. I think a thing to keep in mind that recommenders may be proprietary, and thus pointedly black box. I view interpretability as a conversation between the recommender operator and the user, subject, or regulator of the recommender that is unfortunately mediated by confidentiality. I view explanation as like debugging, something you do in house in dev, and it bewilders me that one would launch a system that one can't explain. But, sometimes we make deployments first and then rationalize them after ...
- dataflow 5y ago
  > There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. Does rationalize here mean "deduce that a supporting argument would exist", or does it mean "actually determine what that argument is"? Like imagine if the brain is solving an approximate (or different) problem first, deducing that there's a solution that's very likely to be feasible for the original problem, then working out all the details afterward. Is that rationalizing after the fact or before the fact?
- akiselev 5y ago
  > There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. The problem with that is we can't simulate human brains to test the integrity of that rationalization. Assuming the algorithm is repeatable, the data from the rationalization can be used to generate adversarial input from the original that tests the conclusion is actually responding to those factors. At the very least, this would give courts an avenue to validate the algorithms in practice.
jerf 5y ago
"The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted" Not for long. This is trivially forgable with current AI tech. It's easy. Your AI tells you that X is in category Y because there's an 80% match on reason 1, a 65% match on reason 2, a 45% match for 3, etc. etc. Reason number 2 is, for the sake of argument, outright racist, and the person running this AI knows that, so they simply hand you an explanation with that reason removed. You have no way of knowing whether this has happened, neither does anyone else, and people continue to accuse your AI of bias. (Especially after I "helpfully" normalize the reason factors for you against the list I handed you, not what came out of the AI.) Any human-comprehensible explanation produced by a program is certainly human-editable, and almost certainly practically editable by code. If you were going to design an AI architecture to provide you parallel construction reasons systematically, it would be hard to produce something better than a neural net.
- tshaddox 5y ago
  Wouldn’t a good explanation need to account for everything it’s purporting to account for and not contain any details that can be omitted or altered without changing what it accounts for? An explanation that omits that reason number 2 would not account for why the output changes when the input to reason number 2 changes in isolation.
  - jerf 5y ago
    You, the person expecting an explanation, won't have access to the model to twiddle with the parameters and see what happens. In this case I'm assuming something like "the bank explaining to you why your loan was accepted/rejected", not just the model explaining to the bank employees why it was accepted/rejected, because this is where the social friction is and why anybody cares at all about explanations.
    - tshaddox 5y ago
      If you have no way of determining whether the explanation is valid then it might as well be a completely made up explanation anyway.

aeternum 5y ago

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. I wonder if we could tell whether or not a sufficiently complex NN or AI were doing the same, or if it even matters. It at least feels somewhat useless if system A makes the decision but then system B comes up with justification (even if the justification is plausible).

troelsSteegin 5y ago
XAI research talks about exactly that, system B "surrogate models" that are consistent with the recommender right around the locus of a prediction. If the surrogate model is easy to interpret, you can forgo directly explaining the base recommender. I think a thing to keep in mind that recommenders may be proprietary, and thus pointedly black box. I view interpretability as a conversation between the recommender operator and the user, subject, or regulator of the recommender that is unfortunately mediated by confidentiality. I view explanation as like debugging, something you do in house in dev, and it bewilders me that one would launch a system that one can't explain. But, sometimes we make deployments first and then rationalize them after ...
dataflow 5y ago
> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. Does rationalize here mean "deduce that a supporting argument would exist", or does it mean "actually determine what that argument is"? Like imagine if the brain is solving an approximate (or different) problem first, deducing that there's a solution that's very likely to be feasible for the original problem, then working out all the details afterward. Is that rationalizing after the fact or before the fact?
akiselev 5y ago
> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. The problem with that is we can't simulate human brains to test the integrity of that rationalization. Assuming the algorithm is repeatable, the data from the rationalization can be used to generate adversarial input from the original that tests the conclusion is actually responding to those factors. At the very least, this would give courts an avenue to validate the algorithms in practice.

jerf 5y ago

"The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted" Not for long. This is trivially forgable with current AI tech. It's easy. Your AI tells you that X is in category Y because there's an 80% match on reason 1, a 65% match on reason 2, a 45% match for 3, etc. etc. Reason number 2 is, for the sake of argument, outright racist, and the person running this AI knows that, so they simply hand you an explanation with that reason removed. You have no way of knowing whether this has happened, neither does anyone else, and people continue to accuse your AI of bias. (Especially after I "helpfully" normalize the reason factors for you against the list I handed you, not what came out of the AI.) Any human-comprehensible explanation produced by a program is certainly human-editable, and almost certainly practically editable by code. If you were going to design an AI architecture to provide you parallel construction reasons systematically, it would be hard to produce something better than a neural net.

tshaddox 5y ago
Wouldn’t a good explanation need to account for everything it’s purporting to account for and not contain any details that can be omitted or altered without changing what it accounts for? An explanation that omits that reason number 2 would not account for why the output changes when the input to reason number 2 changes in isolation.
- jerf 5y ago
  You, the person expecting an explanation, won't have access to the model to twiddle with the parameters and see what happens. In this case I'm assuming something like "the bank explaining to you why your loan was accepted/rejected", not just the model explaining to the bank employees why it was accepted/rejected, because this is where the social friction is and why anybody cares at all about explanations.
  - tshaddox 5y ago
    If you have no way of determining whether the explanation is valid then it might as well be a completely made up explanation anyway.