4 ms·

To me as an AI novice this article seemed like a good overview. It did not address one problem I have with AI however, which is the inherent lack of transparenc

To me as an AI novice this article seemed like a good overview. It did not address one problem I have with AI however, which is the inherent lack of transparency. That is, unlike normal programs, we have an input and an output, but the reasoning in between is a black box to human intelligence. This problem has to be solved before we can turn over any really vital tasks to AI with confidence IMO.

visarga 9y ago
It's not a 'black box'. We know what goes in: matrix multiplication and simple math, in millions of similar units. We can also probe the net to know what a neuron does, or what a specific configuration means - but it might be disappointing - each individual neuron or weight might not have much meaning on its own and even if it were removed, the neural net world work just as well (we can remove almost 95% of the neurons and still make it work). We can also perturb the inputs to study how the outputs depend on them. There are many things we can do to probe a neural net. We might not be able to say for sure what situation will make it fail, but we can try it with millions of tests to see how good it is. That's how neural nets are rated, actually - by testing on loads of new data, they haven't seen during training. On the other hand, human brains are really black boxes. We know much less about how they work or how they reach a conclusion. The fact that we can self-report is not the same as knowing how a human would behave in any reasonable situation. We don't actually know how people will behave. And yet, a few people do have access to incredible power to do harm to humanity. We have had to live with that fear since the invention of atomics and genetic viruses. So I don't think the argument that "neural nets are black boxes" is so powerful. As opposed to what, I would say? Even our dear president is such a black box. We have no idea how it works, or what he will do next.
zEVE16Ug50tV 9y ago
That's a good and deep point. Let's consider the case of writing an algorithm to drive a car. There are some ideas people have had: 1) What really matters is what happens in the worst case, so we need explanations in the worst case, but not necessarily the rest of the time: when the car is choosing a slightly more efficient trajectory, we don't need an explanation of why it 'chose' to do that. In the case of a crash, though, we'd like to know precisely what went wrong. This suggests, perhaps, that we could have a simpler and more transparent fallback system that usually is uninvolved in driving, but that takes 'responsibility' and has the ability to control the vehicle at a minimum level of safety (rather than efficiency). 2) Humans seem to produce make decisions first and explanations later http://www.skepticink.com/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ http://www.skepticink.com/tippling/2013/11/14/post-hoc-ratio...: 'To the question many people ask about politics — Why doesn’t the other side listen to reason? — Haidt replies: We were never designed to listen to reason. When you ask people moral questions, time their responses and scan their brains, their answers and brain activation patterns indicate that they reach conclusions quickly and produce reasons later only to justify what they’ve decided.' so perhaps we could also train an ML agent to produce explanations that we find persuasive?
- stareatgoats 9y ago
  Agree, the need for transparency occurs mainly when something goes wrong. But it is not also a question of control? Without human understanding there is no human control, with no human control we have HAL ... which sort of answers your second point: no, we should not let AI come up with it's own explanations that we just find persuasive.

visarga 9y ago

It's not a 'black box'. We know what goes in: matrix multiplication and simple math, in millions of similar units. We can also probe the net to know what a neuron does, or what a specific configuration means - but it might be disappointing - each individual neuron or weight might not have much meaning on its own and even if it were removed, the neural net world work just as well (we can remove almost 95% of the neurons and still make it work). We can also perturb the inputs to study how the outputs depend on them. There are many things we can do to probe a neural net. We might not be able to say for sure what situation will make it fail, but we can try it with millions of tests to see how good it is. That's how neural nets are rated, actually - by testing on loads of new data, they haven't seen during training. On the other hand, human brains are really black boxes. We know much less about how they work or how they reach a conclusion. The fact that we can self-report is not the same as knowing how a human would behave in any reasonable situation. We don't actually know how people will behave. And yet, a few people do have access to incredible power to do harm to humanity. We have had to live with that fear since the invention of atomics and genetic viruses. So I don't think the argument that "neural nets are black boxes" is so powerful. As opposed to what, I would say? Even our dear president is such a black box. We have no idea how it works, or what he will do next.

zEVE16Ug50tV 9y ago

That's a good and deep point. Let's consider the case of writing an algorithm to drive a car. There are some ideas people have had: 1) What really matters is what happens in the worst case, so we need explanations in the worst case, but not necessarily the rest of the time: when the car is choosing a slightly more efficient trajectory, we don't need an explanation of why it 'chose' to do that. In the case of a crash, though, we'd like to know precisely what went wrong. This suggests, perhaps, that we could have a simpler and more transparent fallback system that usually is uninvolved in driving, but that takes 'responsibility' and has the ability to control the vehicle at a minimum level of safety (rather than efficiency). 2) Humans seem to produce make decisions first and explanations later http://www.skepticink.com/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ http://www.skepticink.com/tippling/2013/11/14/post-hoc-ratio...: 'To the question many people ask about politics — Why doesn’t the other side listen to reason? — Haidt replies: We were never designed to listen to reason. When you ask people moral questions, time their responses and scan their brains, their answers and brain activation patterns indicate that they reach conclusions quickly and produce reasons later only to justify what they’ve decided.' so perhaps we could also train an ML agent to produce explanations that we find persuasive?

stareatgoats 9y ago
Agree, the need for transparency occurs mainly when something goes wrong. But it is not also a question of control? Without human understanding there is no human control, with no human control we have HAL ... which sort of answers your second point: no, we should not let AI come up with it's own explanations that we just find persuasive.