4 ms·

He's not wrong though. We know precisely how any given neural network works, it's just not understandable to us. The point is though, the functions we're tryin

by lars 7y ago

He's not wrong though. We know precisely how any given neural network works, it's just not understandable to us. The point is though, the functions we're trying to learn, like image classifiers, have no responsibility to us to be understandable. In fact, the brightest minds of several generations have worked hard on trying to write down rules to do image classification, and they never came up with anything that works. There is a huge space of functions that are beyond what a human can understand, where we can't write down rules to express the function. This is precisely where we need to use machine learning to find the functions. Lack of explainability is not a bug, the entire point of a neural nets is to find functions that are beyond what we can understand.

opless 7y ago
Lack of explainability is not a bug, it's a wholly missing feature. There are a whole set of mathematical functions that are beyond the range of a typical human mind to understand. But there are proofs that exist that explain how they are correct. I don't think it's a big ask for a new model to be able to justify its own determinations. Of course, that makes things slightly more difficult for all those in research and those selling snake oil and everyone in between. But that's what the difference between science and alchemy is isn't it?
- lars 7y ago
  If I show you a picture and ask you if its a dog or a cat, you can answer the question, but you can't tell me why. You can make up a story to justify why it's a dog or a cat, but if someone tries to implement the procedure you give, it won't work. If it did, we could do computer vision by implementing your justification. You are a black box AI. I can nevertheless trust you to classify dogs vs cats.
  - opless 7y ago
    No. I'm natural general intelligence with no safety features. One that has a hallucinationary view of the world an a neural architecture tracing back a couple of hundred of millions of years that continuously learns and has redundant features, and has the ability of self reflection. We're talking about algorithms that are based on a simplified neural architecture, no redundancy, no self reflection and are still quite immature. Nevertheless we're being asked to trust a black box AI, that you cannot interrogate? Yes, of course, what's the worst that can happen?
    - lars 7y ago
      It may well be that the human visual system is better for most tasks, but then that is what matters, which system is better for the task. The presence of an explanation doesn't matter. Neither system comes with an explanation. On the other hand, if you place Magnus Carlsen against AlphaZero in a game of chess, I will bet on AlphaZero. If however you reduce the complexity of AlphaZero down to a level that it can produce an explanation I can understand, I would instead bet on Magnus Carlsen. Of course we should care about the quality of AI systems, but chasing a human understandable explanation is just the wrong way to go about it, since it in many cases necessarily limits quality of the decisions.
      - opless 7y ago
        I respectably disagree. Here's a paper that you may not have read. https://arxiv.org/pdf/1806.00069.pdf https://arxiv.org/pdf/1806.00069.pdf
      - naresh_xai 7y ago
        Again the non-sequitur argument of an explainable model must be worse than a deep learning system and there has to be a tradeoff. You don’t need to reduce complexity to induce explainability. You just need to decompose the function into smaller parts which you can understand. Contrastive LRP for example is a Function decomposition technique for explaining deep neural networks with high fidelity.
dwaltrip 7y ago
> We know precisely how any given neural network works, it's just not understandable to us. You can't know something if it isn't understandable. Knowledge requires some sort of understanding.
- lars 7y ago
  Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog. That is an exact reproducable procedure that tells you how the neural network works. But you are a human being, and have a short term memory capacity of about 7 items. And 100 million parameters is too much for a human to really understand. The point is that there are strong reasons to believe that no procedure for classifying cat vs dog is small enough that humans can wrap their heads around it. And why is this a problem? The human vision system is exactly the same, complete black box, yet we rely on it every day.
  - opless 7y ago
    No. It isn't exactly the same. It shares some features, I grant you, but decoding cats/dogs by welding a classifier to the equivalent of V3/V4 isn't what a mammal does. Furthermore; A "conscious" short term memory of 7-10 sequences is correct. So we break issues down into manageable chunks and it's turtles all the way down. Comparing the product of >200m years of evolution Vs a decade or so of human endeavour is a strawman.
  - salawat 7y ago
    >Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog. My friend, you have not even scratched the surface. First off, an elucidation of the inputs and the procedure by which an output is generated does not an explanation of the system make. When I look at an image classifier, I want to know what features it's using to make a determination of being cat-like. That way, I can compare that with my own experience to make sure if I cut someone loose with a cat/dog detector, someone doesn't get given the idea that a young bear is a dog. Your trivial AI cat/dog detector may identify cat/dog like features in a still, but that's not equivalent with being able to distill the essence of cat/dog from the reality and common experience of the world around us. If you're going to try to sell me on a system that purportedly knows what something is, I expect it to actually represent the level of intelligence you make it out to possess.The neural networks we manufacture are of a level of magnitude so much narrower than what ML people seem to want the lay person to give them credit for. Think of it this way: As a programmer, I am expected to be able to create an accurate enough representation of what is going on in a complex system that a non-programmer can connect what the system is doing to whether or not it is doing what it should be. Given enough time and patience on the non-programmer's part, I should be able to transfer and walk through enough information where the non-programmer suddenly becomes a novice programmer because they have had the same foundational skill and knowledge structures communicated to them. No one will be satisfied with "I chucked this data in, therefore it's a cat/dog detector now. No more questions." Especially when you start applying that to decisions of life-altering importance. You must be 10% smarter than the piece of equipment for it to be lynchpin in a life-critical application. That means being able to explain what your system does, how changes to inputs will effect it, what it's error margins are, what the safe operating conditions are, when it's plain flat out wrong, and as much as possible, why. Until such knowledge can be sufficiently communicated, I see no reason to take even the most well-known luminary trying to handwave explicability as anything but trying to avoid having to uncover enough of the mystery of what they are working on in order to meet what has been accepted as sufficient due diligence. To do so is patently unwise, and implicitly accepts far more egg breaking to make an omelette than we (those whose lives will be in the system's hands) should be willing to entertain.

by lars 7y ago

opless 7y ago

Lack of explainability is not a bug, it's a wholly missing feature. There are a whole set of mathematical functions that are beyond the range of a typical human mind to understand. But there are proofs that exist that explain how they are correct. I don't think it's a big ask for a new model to be able to justify its own determinations. Of course, that makes things slightly more difficult for all those in research and those selling snake oil and everyone in between. But that's what the difference between science and alchemy is isn't it?

lars 7y ago
If I show you a picture and ask you if its a dog or a cat, you can answer the question, but you can't tell me why. You can make up a story to justify why it's a dog or a cat, but if someone tries to implement the procedure you give, it won't work. If it did, we could do computer vision by implementing your justification. You are a black box AI. I can nevertheless trust you to classify dogs vs cats.
- opless 7y ago
  No. I'm natural general intelligence with no safety features. One that has a hallucinationary view of the world an a neural architecture tracing back a couple of hundred of millions of years that continuously learns and has redundant features, and has the ability of self reflection. We're talking about algorithms that are based on a simplified neural architecture, no redundancy, no self reflection and are still quite immature. Nevertheless we're being asked to trust a black box AI, that you cannot interrogate? Yes, of course, what's the worst that can happen?
  - lars 7y ago
    It may well be that the human visual system is better for most tasks, but then that is what matters, which system is better for the task. The presence of an explanation doesn't matter. Neither system comes with an explanation. On the other hand, if you place Magnus Carlsen against AlphaZero in a game of chess, I will bet on AlphaZero. If however you reduce the complexity of AlphaZero down to a level that it can produce an explanation I can understand, I would instead bet on Magnus Carlsen. Of course we should care about the quality of AI systems, but chasing a human understandable explanation is just the wrong way to go about it, since it in many cases necessarily limits quality of the decisions.
    - opless 7y ago
      I respectably disagree. Here's a paper that you may not have read. https://arxiv.org/pdf/1806.00069.pdf https://arxiv.org/pdf/1806.00069.pdf
    - naresh_xai 7y ago
      Again the non-sequitur argument of an explainable model must be worse than a deep learning system and there has to be a tradeoff. You don’t need to reduce complexity to induce explainability. You just need to decompose the function into smaller parts which you can understand. Contrastive LRP for example is a Function decomposition technique for explaining deep neural networks with high fidelity.

dwaltrip 7y ago

> We know precisely how any given neural network works, it's just not understandable to us. You can't know something if it isn't understandable. Knowledge requires some sort of understanding.

lars 7y ago
Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog. That is an exact reproducable procedure that tells you how the neural network works. But you are a human being, and have a short term memory capacity of about 7 items. And 100 million parameters is too much for a human to really understand. The point is that there are strong reasons to believe that no procedure for classifying cat vs dog is small enough that humans can wrap their heads around it. And why is this a problem? The human vision system is exactly the same, complete black box, yet we rely on it every day.
- opless 7y ago
  No. It isn't exactly the same. It shares some features, I grant you, but decoding cats/dogs by welding a classifier to the equivalent of V3/V4 isn't what a mammal does. Furthermore; A "conscious" short term memory of 7-10 sequences is correct. So we break issues down into manageable chunks and it's turtles all the way down. Comparing the product of >200m years of evolution Vs a decade or so of human endeavour is a strawman.
- salawat 7y ago
  >Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog. My friend, you have not even scratched the surface. First off, an elucidation of the inputs and the procedure by which an output is generated does not an explanation of the system make. When I look at an image classifier, I want to know what features it's using to make a determination of being cat-like. That way, I can compare that with my own experience to make sure if I cut someone loose with a cat/dog detector, someone doesn't get given the idea that a young bear is a dog. Your trivial AI cat/dog detector may identify cat/dog like features in a still, but that's not equivalent with being able to distill the essence of cat/dog from the reality and common experience of the world around us. If you're going to try to sell me on a system that purportedly knows what something is, I expect it to actually represent the level of intelligence you make it out to possess.The neural networks we manufacture are of a level of magnitude so much narrower than what ML people seem to want the lay person to give them credit for. Think of it this way: As a programmer, I am expected to be able to create an accurate enough representation of what is going on in a complex system that a non-programmer can connect what the system is doing to whether or not it is doing what it should be. Given enough time and patience on the non-programmer's part, I should be able to transfer and walk through enough information where the non-programmer suddenly becomes a novice programmer because they have had the same foundational skill and knowledge structures communicated to them. No one will be satisfied with "I chucked this data in, therefore it's a cat/dog detector now. No more questions." Especially when you start applying that to decisions of life-altering importance. You must be 10% smarter than the piece of equipment for it to be lynchpin in a life-critical application. That means being able to explain what your system does, how changes to inputs will effect it, what it's error margins are, what the safe operating conditions are, when it's plain flat out wrong, and as much as possible, why. Until such knowledge can be sufficiently communicated, I see no reason to take even the most well-known luminary trying to handwave explicability as anything but trying to avoid having to uncover enough of the mystery of what they are working on in order to meet what has been accepted as sufficient due diligence. To do so is patently unwise, and implicitly accepts far more egg breaking to make an omelette than we (those whose lives will be in the system's hands) should be willing to entertain.