6 ms·

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the com

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interprets data from the components. If there was a crash, they can tie it down to "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" without needing much interpretability of the neural network directly. If it's an end-to-end system, such as the nvidia car which connects the camera inputs directly to a wheel output, then explainability of that network is much more important. There was a poll I saw on twitter a while ago (can't seem to find now) that had asked if users would prefer to be treated by an AI doctor with low explainability but 99% accuracy, or high explainability but only 70% accuracy. I think context is key here as well. If the expert doctor working in tandem with the explainable system is better than the non-explainable system, you'd want the explainable system. But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

kazinator 9y ago
> explainability is necessary to use it. For that matter, freedom from daily blue screens and reboots is not needed to use an operating system. All sorts of non-AI software has confusing states and quirks, yet is in wide use.
ghaff 9y ago
>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system. Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so." Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.
- chrisfosterelli 9y ago
  Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?
  - ghaff 9y ago
    Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat. If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.
- marcosdumay 9y ago
  The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.
mseebach 9y ago
There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place. Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.
- throwaway2048 9y ago
  If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
  - mseebach 9y ago
    Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.
    - throwaway2048 9y ago
      but you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up
    - twblalock 9y ago
      What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.
      - mseebach 9y ago
        It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.
deleted 9y ago
[deleted]
kurthr 9y ago
I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition. Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?
moseandre 9y ago
>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. I have to be able to debug anything I work with; debugging is necessary. >> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.
- gmueckl 9y ago
  I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).
  - darepublic 9y ago
    It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve
    - gmueckl 9y ago
      Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent. I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them. Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.
pacala 9y ago
The poll is biased. There is no evidence that performance and explainability are at odds.
- chrisfosterelli 9y ago
  With our current models they definitely are at odds. There is a strong correlation between higher average performance and lower explainability when comparing machine learning approaches like neural nets, linear models, decision trees, etc for nearly all tasks.
PopePompous 9y ago
I cannot explain how I form the sentences I speak. I still find speaking useful.

kazinator 9y ago

> explainability is necessary to use it. For that matter, freedom from daily blue screens and reboots is not needed to use an operating system. All sorts of non-AI software has confusing states and quirks, yet is in wide use.

ghaff 9y ago

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system. Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so." Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

chrisfosterelli 9y ago
Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?
- ghaff 9y ago
  Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat. If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.
marcosdumay 9y ago
The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.

mseebach 9y ago

There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place. Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.

throwaway2048 9y ago
If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
- mseebach 9y ago
  Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.
  - throwaway2048 9y ago
    but you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up
  - twblalock 9y ago
    What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.
    - mseebach 9y ago
      It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.

deleted 9y ago

[deleted]

kurthr 9y ago

I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition. Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?

moseandre 9y ago

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. I have to be able to debug anything I work with; debugging is necessary. >> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.

gmueckl 9y ago
I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).
- darepublic 9y ago
  It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve
  - gmueckl 9y ago
    Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent. I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them. Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.

pacala 9y ago

The poll is biased. There is no evidence that performance and explainability are at odds.

chrisfosterelli 9y ago
With our current models they definitely are at odds. There is a strong correlation between higher average performance and lower explainability when comparing machine learning approaches like neural nets, linear models, decision trees, etc for nearly all tasks.

PopePompous 9y ago

I cannot explain how I form the sentences I speak. I still find speaking useful.