"To prevent the system from doing anything harmful or unethical, it may be necessary to challenge it to explain the logic for a particular action. That logic mi

"To prevent the system from doing anything harmful or unethical, it may be necessary to challenge it to explain the logic for a particular action. That logic might be too complex for a person to comprehend, so the researchers suggest having another AI debate the wisdom of..." Very Westworld :) What this made me think of is "rationalizations" in humans. Rationalizations are how we* "explain" (and probably also think about) complex problems/ For example, you might give a human the complex task of playing cricket. They gain experience and become proficient. The person is then asked why they made some decision. The response will be a rationalization, a simple explanation built from simple truths about cricket, cricket theory (if they know any), this match, that player, that play, the score & such. This is mostly a lie, usually. The truth is that the decision was not made by feel, not simple rationalization. It is not the real "why." The real decisions were made by "instinct^," a much more complex decision making framework. Instinct (acquired instinct) is built from countless hours of experience playing cricket and just generally being human. The experienced player "just knows" which way the ball is going to go and starts running. He is interpretting (instinctively) the ballistic trajectory of the ball, the sound it made when it hit the bat, the reactions of other players, prior knowledge about the batsman... This is too much information, too fast witt too much resolution for a rationalization. This much information can only be processed "subconciously" in humans, with rationalizations constructed post facto. Yet... humans still have rationalizations. My first question is "why?" Why bother with these slow, clunky & fraudulent rationalizations? Do they play a role in real-time thinking at all? Is our ability to rationalize only there to tell the manager we did X because Y? Both? My 2c is that I think there is an important interaction layer between simple rationalizations that we can explain to one another and the complex rationality of our subconcious. Both affect the way the other works, yet they are still somewhat distinct. Interesting research area. Good luck to the researchers. I hope you find something good :) ^ What made me think along these lines is an "intelligence vs consiousness" definition framework favoured by Yuval Harari (historian/philospher, not a CS guy) and others. Intelligence = ability to perform tasks ..involving decision making. Consiousness = the ability to *feel*, ..in a Benthamite pleasure/pain sense. He agrees that most human intelligence works via feelings, but doesn't expect machine intelligence to be bundled in this way. Not sure I agree. It may not be possible to unbundle feeling from intelligence in machines. That said, we people do undeniably have the abilty to rationalize as well as feel. In any case, I thought a similar yet subtly different dichotomy might work better. Theoried vs theory-less machine intelligence. Thus far, statistical ML techniques produce mostly theory-less machines. They can make good decisions but don't "know" why. All "feelings" no rationalization. By forcing the machine to communicate a theory (a simplification or rationalization), the machine will have to produce a theory. Once it has a rationalization/theory, the machine can use it to augment its theory-less logic, the logic that produced the theory. Any disagreement between theory and theory-less (rationality vs feeling) results in a choice from the following: (1) update the theory, (2) change the decision, (3) tolerate some level of cognitive dissonance. This is too westworld so let me try it in different terms...: (Step 1) An ML machine observes data generated by a complex function. For any given W, X & Z it predicts Y. (Step 2) The ML machine must communicate why it predicted a specific Y. The output is a function, an aproximation of the underlying function generating the observations. (step 3 - underpants step) ... Now that the machine has theories, it can test its own ML-based decisions for theoretical consistency. QAgreements strengthen the theory. Disagreements cause cognitive dissonance, loss of feeling or rationality, madness... crap! I'm back in wetsworld. dammit.