3 ms·

So they used a symbolic planner to direct the robot's actions and showed this to the human participants. They didn't use a neural network in the learning proces

by pharke 7y ago

So they used a symbolic planner to direct the robot's actions and showed this to the human participants. They didn't use a neural network in the learning process. So how does this advance the explainability of the hard to explain parts of AI? Better yet, how would this strategy even be applicable to NN based learning models? We don't even have sufficient knowledge of how our own symbolic systems map to our brain activity. There is even evidence against a direct causal relationship between our symbolic narrative and our actions such as split brain experiments where the actions of the nonverbal hemisphere are "explained" or rationalized by the other post hoc. See also postdictive illusions and the varied arguments against free will. My view is that the processes that constitute our minds are actually almost entirely non-verbal and the language oriented parts are merely a specialization of a more general rule. I'm not saying that language has no effect on our conscious processes, it wouldn't be a useful system if it played no part, but I do consider it to be more of a side channel or perhaps an abstraction of the processes that are driving what we call will or thought. I believe this may be a fundamental flaw in how we approach understanding consciousness or any other aspect of intelligence.

timkam 7y ago
I understand your concerns, but there is a large body of XAI research that addresses them. For example, there are post-hoc explainability algorithms like LIME (https://github.com/marcotcr/lime https://github.com/marcotcr/lime) that "explain" the classification decisions of machine learning models. The obvious alternative is to use (deep) learning only to mine simpler, fully explainable models (sacrifice correlation power for intelligibility). What I don't think exist yet are approaches that go full scale with the "reasoning backwards" approach. Like a neural network making a BS classification decision and then trying to "reason backwards" in a symbolic way to defend the decision. But works like this one: https://openai.com/blog/debate/ https://openai.com/blog/debate/ go roughly into this direction (the work seemed to be, at the time of writing, in its infancy).

by pharke 7y ago

timkam 7y ago

I understand your concerns, but there is a large body of XAI research that addresses them. For example, there are post-hoc explainability algorithms like LIME (https://github.com/marcotcr/lime https://github.com/marcotcr/lime) that "explain" the classification decisions of machine learning models. The obvious alternative is to use (deep) learning only to mine simpler, fully explainable models (sacrifice correlation power for intelligibility). What I don't think exist yet are approaches that go full scale with the "reasoning backwards" approach. Like a neural network making a BS classification decision and then trying to "reason backwards" in a symbolic way to defend the decision. But works like this one: https://openai.com/blog/debate/ https://openai.com/blog/debate/ go roughly into this direction (the work seemed to be, at the time of writing, in its infancy).