42 ms·

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. I wonder if we could tell whether or not a

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. I wonder if we could tell whether or not a sufficiently complex NN or AI were doing the same, or if it even matters. It at least feels somewhat useless if system A makes the decision but then system B comes up with justification (even if the justification is plausible).

troelsSteegin 6y ago
XAI research talks about exactly that, system B "surrogate models" that are consistent with the recommender right around the locus of a prediction. If the surrogate model is easy to interpret, you can forgo directly explaining the base recommender. I think a thing to keep in mind that recommenders may be proprietary, and thus pointedly black box. I view interpretability as a conversation between the recommender operator and the user, subject, or regulator of the recommender that is unfortunately mediated by confidentiality. I view explanation as like debugging, something you do in house in dev, and it bewilders me that one would launch a system that one can't explain. But, sometimes we make deployments first and then rationalize them after ...
dataflow 6y ago
> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. Does rationalize here mean "deduce that a supporting argument would exist", or does it mean "actually determine what that argument is"? Like imagine if the brain is solving an approximate (or different) problem first, deducing that there's a solution that's very likely to be feasible for the original problem, then working out all the details afterward. Is that rationalizing after the fact or before the fact?
akiselev 6y ago
> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. The problem with that is we can't simulate human brains to test the integrity of that rationalization. Assuming the algorithm is repeatable, the data from the rationalization can be used to generate adversarial input from the original that tests the conclusion is actually responding to those factors. At the very least, this would give courts an avenue to validate the algorithms in practice.

troelsSteegin 6y ago

XAI research talks about exactly that, system B "surrogate models" that are consistent with the recommender right around the locus of a prediction. If the surrogate model is easy to interpret, you can forgo directly explaining the base recommender. I think a thing to keep in mind that recommenders may be proprietary, and thus pointedly black box. I view interpretability as a conversation between the recommender operator and the user, subject, or regulator of the recommender that is unfortunately mediated by confidentiality. I view explanation as like debugging, something you do in house in dev, and it bewilders me that one would launch a system that one can't explain. But, sometimes we make deployments first and then rationalize them after ...

dataflow 6y ago

> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. Does rationalize here mean "deduce that a supporting argument would exist", or does it mean "actually determine what that argument is"? Like imagine if the brain is solving an approximate (or different) problem first, deducing that there's a solution that's very likely to be feasible for the original problem, then working out all the details afterward. Is that rationalizing after the fact or before the fact?

akiselev 6y ago

> There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after. The problem with that is we can't simulate human brains to test the integrity of that rationalization. Assuming the algorithm is repeatable, the data from the rationalization can be used to generate adversarial input from the original that tests the conclusion is actually responding to those factors. At the very least, this would give courts an avenue to validate the algorithms in practice.