5 ms·

'Explainable Artificial Intelligence': Cracking Open the Black Box of AI

57 points by sherm8n 10y ago

yummyfajitas 10y ago
Similarly, I feel that a car shouldn't drive too fast. If it does drive too fast then a human running after it might be unable to catch up!
- parenthephobia 10y ago
  This seems flippant. If a car is fast we generally understand why. We don't need to worry that under some rarely-encountered combination of circumstances it will unexpectedly do a handbrake turn and open the fuel cap.
- brudgers 10y ago
  Speed limits on public roads place legal limits on the operation of automobiles. Rules in formula racing place limits on the speed which race cars can travel.a
harperlee 10y ago
Newbie question: I've heard that PGMs are a superset of neural networks. In PGM materials that's I've read, the topology of the networks shown as example is made of node that are manually chosen and represent concept (smart student, good grades, difficult subject, etc.). Whereas a neural network example is usually a huge set of nodes that end up finding their meaning on their own. I also vaguely recall a tutorial in which you can highlight the nodes that contributed to the classification - the only thing is that they don't have meaning for a human. Then when the article states: > restrict the way nodes in a neural network consider things to ‘concepts’ like colour and shapes and textures. Aren't these just PGMs? Are they NNs? Is it just a methodology approach on how to select the topology? Don't you lose the automatic meaning / structure search? I'm a little bit confused...
- eli_gottlieb 10y ago
  PGMs and neural networks are usually two completely different things. Neural networks involve graphs, but those graphs usually represent continuous relaxations of circuits -- rather than probability models.
- imh 10y ago
  PGMs are interesting in how they represent distributions over the values of their nodes. In neural networks, (for the most part) those nodes are deterministic, so from a PGM perspective the distribution is trivial (up until the final output prediction). Performing inference in a neural net with stochastic nodes would be crazy hard, so the best you can do is usually MC with some kind of reparametrization trick to keep your gradients around.
bencollier49 10y ago
There's a hell of a lot of money to be made by the person who cracks this. The major blockers preventing a lot of AI being rolled out across the EU are laws which stipulate that you have to be able to explain a decision to, for example, refuse a person credit. Not to mention the fact that we can correct faulty assumptions on the fly if we can get the networks to introspect.
- sdenton4 10y ago
  Meh. Why not just use a decision tree for the credit decision? Perfectly explainable, and the feature domain isn't as difficult (and full of symmetry) as vision or language problems.
  - PeterisP 10y ago
    Well, a good way to use decision trees for that is random forests, and you're right back to something that's not really easily explainable.
bko 10y ago
I think the author is overstating the importance of being able to explain in human terms decisions made by a neural network. For instance , there is no one reason that I am able to recognize a dog as such. Any feature or combination of features I can think of can be had in another animal. Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me. The examples normally given for wildly inaccurate predictions were concocted by training a separate neural network to trick the original neural network which seems be just showcasing the effectiveness of neural networks rather than highlighting a weakness. Also, I would note that human intuition is not immune to tricks. For instance optical illusions regularly trick our perception.
- akyu 10y ago
  >Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me. There is an interesting experiment you can do to directly demonstrate this to yourself From Julian Jaynes' The Origin of Consciousness in the Breakdown of the Bicameral Mind > Take any two unequal objects, such as a pen and pencil or two unequally filled glasses of water, and place them on the desk in front of you. Then, partly closing your eyes to increase your attention to the task, pick up each one with the thumb and forefinger and judge which is heavier. Now introspect on everything you are doing. You will find yourself conscious of the feel of the objects against the skin of your fingers, conscious of the slight downward pressure as you feel the weight of each, conscious of any protuberances on the sides of the objects, and so forth. And now the actual judging of which is heavier. Where is that? Lo! the very act of judgment that one object is heavier than the other is not conscious. It is somehow just given to you by your nervous system. If we call that process of judgment thinking, we are finding that such thinking is not conscious at all. A simple experiment, yes, but extremely important. It demolishes at once the entire tradition that such thought processes are the structure of the conscious mind.
- mwexler 10y ago
  No, the importance is massive, and you understate it. While we may all want to just believe, the regulatory ecosystem worldwide (where it applies, esp. in financial realm) demands that we provide explanations of why models made certain decisions on certain data. Without this ability, models will not be allowed to drive innovation or decisions in many areas of life, from financial (credit and risk) to medical (recommendations for treatment) to legal (best contract approach or best defense approach for a lawsuit). Saying that humans make mistakes and cannot explain their decisions is, in fact, one of the very reasons we want to have better models. We hope they will do better than most people to create a better world. And their explanations will hopefully provide insight into how we as people make good (and bad) decisions. And sure, we can change the laws over time, but having advanced models which can allow humans to understand decisions and even provide diagnostics to improve the models will be transformative. Until then, we will see massive impact in some areas of our lives, and frustrating holdbacks in others, driven either by the distraction of building for regulatory constraints or by choosing not to build in regulated areas at all.
TeMPOraL 10y ago
One issue I don't see considered is - how to ensure that explainable artificial intelligence doesn't lie? Right now, it may not be an issue, but as AI systems get complex ("smart") enough, one need to be sure that the introspective output isn't crafted to influence people looking at it.
- vitiral 10y ago
  Right now it looks like it's being used more as a "debugging" output to make more intelligent Al's. Once they can lie, we will have achieved that goal...
cr0sh 10y ago
Let's say this is possible. How would we know that it (the AI) isn't doing a post-hoc rationalization, or just outright lying about its reasoning? In other words, why do we trust humans more than machines? In fact, why do we not think of humans as machines - just ones made out of different materials? Why do we have this bias that machines are and must-be deterministic, and since humans aren't, they must not be machines? Furthermore, since we know that these AI models are sometimes stochastic, why do we still insist that they be explainable; when humans exhibit the same kind of output, we don't insist upon their determinism...? I'm not certain that we can make these models - especially complex deep-learning CNNs and others like them - explainable, any more than an individual can tell you how his or her brain came up with the solution; most of the time, we employ post-hoc reasoning to explain our decisions, depending on how the output resolves. That - or we lie. Rarely do we say "I don't know" - because to do so is to admit a form of failure. Not admitting such is what helps religion continue, because when we don't know, we can ascribe the reason to another external force instead. If we would just be willing to say "I don't know - but let's try to find out" (insert XKCD here), we might be better off as a species. I don't think an AI model will be any different - or can be. If we insist on having an AI be able to deterministically and truthfully tell us exactly how it arrived at such a conclusion, we must be ready to accept that we should do the same with human reasoning as well. Anything less would be hypocritical at best.

57 points by sherm8n 10y ago

Similarly, I feel that a car shouldn't drive too fast. If it does drive too fast then a human running after it might be unable to catch up!

parenthephobia 10y ago
This seems flippant. If a car is fast we generally understand why. We don't need to worry that under some rarely-encountered combination of circumstances it will unexpectedly do a handbrake turn and open the fuel cap.
brudgers 10y ago
Speed limits on public roads place legal limits on the operation of automobiles. Rules in formula racing place limits on the speed which race cars can travel.a

harperlee 10y ago

Newbie question: I've heard that PGMs are a superset of neural networks. In PGM materials that's I've read, the topology of the networks shown as example is made of node that are manually chosen and represent concept (smart student, good grades, difficult subject, etc.). Whereas a neural network example is usually a huge set of nodes that end up finding their meaning on their own. I also vaguely recall a tutorial in which you can highlight the nodes that contributed to the classification - the only thing is that they don't have meaning for a human. Then when the article states: > restrict the way nodes in a neural network consider things to ‘concepts’ like colour and shapes and textures. Aren't these just PGMs? Are they NNs? Is it just a methodology approach on how to select the topology? Don't you lose the automatic meaning / structure search? I'm a little bit confused...

eli_gottlieb 10y ago
PGMs and neural networks are usually two completely different things. Neural networks involve graphs, but those graphs usually represent continuous relaxations of circuits -- rather than probability models.
imh 10y ago
PGMs are interesting in how they represent distributions over the values of their nodes. In neural networks, (for the most part) those nodes are deterministic, so from a PGM perspective the distribution is trivial (up until the final output prediction). Performing inference in a neural net with stochastic nodes would be crazy hard, so the best you can do is usually MC with some kind of reparametrization trick to keep your gradients around.

bencollier49 10y ago

There's a hell of a lot of money to be made by the person who cracks this. The major blockers preventing a lot of AI being rolled out across the EU are laws which stipulate that you have to be able to explain a decision to, for example, refuse a person credit. Not to mention the fact that we can correct faulty assumptions on the fly if we can get the networks to introspect.

sdenton4 10y ago
Meh. Why not just use a decision tree for the credit decision? Perfectly explainable, and the feature domain isn't as difficult (and full of symmetry) as vision or language problems.
- PeterisP 10y ago
  Well, a good way to use decision trees for that is random forests, and you're right back to something that's not really easily explainable.

bko 10y ago

I think the author is overstating the importance of being able to explain in human terms decisions made by a neural network. For instance , there is no one reason that I am able to recognize a dog as such. Any feature or combination of features I can think of can be had in another animal. Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me. The examples normally given for wildly inaccurate predictions were concocted by training a separate neural network to trick the original neural network which seems be just showcasing the effectiveness of neural networks rather than highlighting a weakness. Also, I would note that human intuition is not immune to tricks. For instance optical illusions regularly trick our perception.

akyu 10y ago
>Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me. There is an interesting experiment you can do to directly demonstrate this to yourself From Julian Jaynes' The Origin of Consciousness in the Breakdown of the Bicameral Mind > Take any two unequal objects, such as a pen and pencil or two unequally filled glasses of water, and place them on the desk in front of you. Then, partly closing your eyes to increase your attention to the task, pick up each one with the thumb and forefinger and judge which is heavier. Now introspect on everything you are doing. You will find yourself conscious of the feel of the objects against the skin of your fingers, conscious of the slight downward pressure as you feel the weight of each, conscious of any protuberances on the sides of the objects, and so forth. And now the actual judging of which is heavier. Where is that? Lo! the very act of judgment that one object is heavier than the other is not conscious. It is somehow just given to you by your nervous system. If we call that process of judgment thinking, we are finding that such thinking is not conscious at all. A simple experiment, yes, but extremely important. It demolishes at once the entire tradition that such thought processes are the structure of the conscious mind.
mwexler 10y ago
No, the importance is massive, and you understate it. While we may all want to just believe, the regulatory ecosystem worldwide (where it applies, esp. in financial realm) demands that we provide explanations of why models made certain decisions on certain data. Without this ability, models will not be allowed to drive innovation or decisions in many areas of life, from financial (credit and risk) to medical (recommendations for treatment) to legal (best contract approach or best defense approach for a lawsuit). Saying that humans make mistakes and cannot explain their decisions is, in fact, one of the very reasons we want to have better models. We hope they will do better than most people to create a better world. And their explanations will hopefully provide insight into how we as people make good (and bad) decisions. And sure, we can change the laws over time, but having advanced models which can allow humans to understand decisions and even provide diagnostics to improve the models will be transformative. Until then, we will see massive impact in some areas of our lives, and frustrating holdbacks in others, driven either by the distraction of building for regulatory constraints or by choosing not to build in regulated areas at all.

TeMPOraL 10y ago

One issue I don't see considered is - how to ensure that explainable artificial intelligence doesn't lie? Right now, it may not be an issue, but as AI systems get complex ("smart") enough, one need to be sure that the introspective output isn't crafted to influence people looking at it.

vitiral 10y ago
Right now it looks like it's being used more as a "debugging" output to make more intelligent Al's. Once they can lie, we will have achieved that goal...

cr0sh 10y ago

Let's say this is possible. How would we know that it (the AI) isn't doing a post-hoc rationalization, or just outright lying about its reasoning? In other words, why do we trust humans more than machines? In fact, why do we not think of humans as machines - just ones made out of different materials? Why do we have this bias that machines are and must-be deterministic, and since humans aren't, they must not be machines? Furthermore, since we know that these AI models are sometimes stochastic, why do we still insist that they be explainable; when humans exhibit the same kind of output, we don't insist upon their determinism...? I'm not certain that we can make these models - especially complex deep-learning CNNs and others like them - explainable, any more than an individual can tell you how his or her brain came up with the solution; most of the time, we employ post-hoc reasoning to explain our decisions, depending on how the output resolves. That - or we lie. Rarely do we say "I don't know" - because to do so is to admit a form of failure. Not admitting such is what helps religion continue, because when we don't know, we can ascribe the reason to another external force instead. If we would just be willing to say "I don't know - but let's try to find out" (insert XKCD here), we might be better off as a species. I don't think an AI model will be any different - or can be. If we insist on having an AI be able to deterministically and truthfully tell us exactly how it arrived at such a conclusion, we must be ready to accept that we should do the same with human reasoning as well. Anything less would be hypocritical at best.