3 ms·

This seems... incorrect. First of all, deep learning absolutely is used in mission critical systems: https://www.techrepublic.com/article/intel-and-ge-healthca

This seems... incorrect. First of all, deep learning absolutely is used in mission critical systems: https://www.techrepublic.com/article/intel-and-ge-healthcares-x-ray-machine-uses-embedded-ai-to-prioritize-scans/ https://www.techrepublic.com/article/intel-and-ge-healthcare... Second, simply because designers use systems that are formally interpretable, that doesn't mean that designers know exactly what causes a given response in the model. Formal interpretability means you know what function the model is approximating -- there's a long way from that to a human-readable interpretation. Finally, even if the designers can tell you what features cause a certain response in the model, they can only tell you that for features that are encoded in the data! There are plenty of features that aren't encoded in your data that can nevertheless affect the outcome of a model!

OnlyOneCannolo 6y ago
I don't think the Intel/GE tool counts as mission critical. There are already radiologists reviewing scans. The tool just flags features they may have missed. Remove the tool, and you're back to business as usual. I don't know much about ML. Are you saying that we don't know how to interpret any ML models at all? Regarding your final point, isn't that true with or without ML? Any mission-critical design process should scrutinize the solution to see if it's complete enough and correct enough.
- thylacine222 6y ago
  That tool has the potential to systematically reduce the likelihood that a certain disease/certain class of people receives extra scrutiny from a radiologist. That's maybe not mission critical on a small scale, but on a large scale it absolutely is. Deep learning models are black box models, which means they are not formally interpretable. You can still sometimes get interpretations out of these models using various methods, but they are fairly underdeveloped, and without actual theories of neural network behavior I don't see that improving anytime soon. You're right, it's true with or without ML. In fact, it's true with human-run systems, too. Consider a police officer who is more likely to pull people over in a certain neighborhood, and that neighborhood was 95% AfAm. IF they were asked why they pulled over more people in that neighborhood, they could say they were discriminating against the neighborhood, which, in and of itself is not racial discrimination. Of course, further inspection of that police officer's records could show that they are biased towards neighborhoods with a high AfAm population, which would be racial discrimination. The same scenario can easily arise in an ML context, but interrogating a machine is a very different context from interrogating a human. First of all, people believe that computers are innately 'unbiased' because they are computers, so making the case that an algorithm is biased is already more difficult. Second, going back to the point I made before -- interpreting a model is not the same as providing a human-readable explanation. Asking a question about racial bias in a model which doesn't even encode for race (as many do, in an ill-conceived attempt to be 'neutral') requires skilled people to understand how to ask the question and how to interpret the answer. There's no plug and play process that one can follow to "scrutinize the solution to see if it's complete enough and correct enough".
- throwaway09223 6y ago
  > The tool just flags features they may have missed. This is how it functions at first. Fast forward a few decades and you'll have doctors saying "The computer model flagged this spot as a concern. Our human review can't find anything, but we know that studies show the computer model has a 95% confidence in locating problems so we recommend surgery anyway. Surgery is lower risk than assuming the model is wrong." Tool dependency changes over time.
  - OnlyOneCannolo 6y ago
    I agree that it will eventually become mission critical. Until then, they have some time to make the implementation interpretable.

OnlyOneCannolo 6y ago

I don't think the Intel/GE tool counts as mission critical. There are already radiologists reviewing scans. The tool just flags features they may have missed. Remove the tool, and you're back to business as usual. I don't know much about ML. Are you saying that we don't know how to interpret any ML models at all? Regarding your final point, isn't that true with or without ML? Any mission-critical design process should scrutinize the solution to see if it's complete enough and correct enough.

thylacine222 6y ago
That tool has the potential to systematically reduce the likelihood that a certain disease/certain class of people receives extra scrutiny from a radiologist. That's maybe not mission critical on a small scale, but on a large scale it absolutely is. Deep learning models are black box models, which means they are not formally interpretable. You can still sometimes get interpretations out of these models using various methods, but they are fairly underdeveloped, and without actual theories of neural network behavior I don't see that improving anytime soon. You're right, it's true with or without ML. In fact, it's true with human-run systems, too. Consider a police officer who is more likely to pull people over in a certain neighborhood, and that neighborhood was 95% AfAm. IF they were asked why they pulled over more people in that neighborhood, they could say they were discriminating against the neighborhood, which, in and of itself is not racial discrimination. Of course, further inspection of that police officer's records could show that they are biased towards neighborhoods with a high AfAm population, which would be racial discrimination. The same scenario can easily arise in an ML context, but interrogating a machine is a very different context from interrogating a human. First of all, people believe that computers are innately 'unbiased' because they are computers, so making the case that an algorithm is biased is already more difficult. Second, going back to the point I made before -- interpreting a model is not the same as providing a human-readable explanation. Asking a question about racial bias in a model which doesn't even encode for race (as many do, in an ill-conceived attempt to be 'neutral') requires skilled people to understand how to ask the question and how to interpret the answer. There's no plug and play process that one can follow to "scrutinize the solution to see if it's complete enough and correct enough".
throwaway09223 6y ago
> The tool just flags features they may have missed. This is how it functions at first. Fast forward a few decades and you'll have doctors saying "The computer model flagged this spot as a concern. Our human review can't find anything, but we know that studies show the computer model has a 95% confidence in locating problems so we recommend surgery anyway. Surgery is lower risk than assuming the model is wrong." Tool dependency changes over time.
- OnlyOneCannolo 6y ago
  I agree that it will eventually become mission critical. Until then, they have some time to make the implementation interpretable.