5 ms·

> It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require

> It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require trust. One way of establishing trust is to explain or prove why the system works. This cannot be done with machine learning. Say that you already have a safe system. Now you make a change. How do you demonstrate to a skeptical audience that the new system is safe without releasing it and counting the bodies?

m0zg 8y ago
But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a story just the other day where doctors were injecting people with some proteins linked to Alzheimer's? They weren't required to be "proven that they're safe" before they did that.
- oldgradstudent 8y ago
  We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on public roads, you need to convince other, possibly skeptical, people that your technology works better than humans. How do you do that with AI/ML-based systems?
  - m0zg 8y ago
    >> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happen.
    - oldgradstudent 8y ago
      Imagine for a moment that you have a working self-driving car in the field, and now you want to make a relatively minor change to the system. How do you convince yourself or a regulator, not that the technology in general is safe, but that your specific change does not make it unsafe? What happens if this specific change increases the fatality rate from 1 in 100 million miles to 1 in 50 million miles? Crashes will still be rare and there's enough statistical noise that when you finally understand the change is bad, you may have killed hundreds.
    - TeMPOraL 8y ago
      > Same as with humans: from statistics. I don't see any other way. I.e. from body count. One problem here is that we know humans have common, stable brain architecture, so the limits and failure modes we experienced are stable too, and can be accounted for and worked around. People won't fail you in completely surprising ways. DNNs are each a different breed; between changing architectures and tuning hyperparameters, I don't see how trust in one instance can be translated to trust un another.

m0zg 8y ago

But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a story just the other day where doctors were injecting people with some proteins linked to Alzheimer's? They weren't required to be "proven that they're safe" before they did that.

oldgradstudent 8y ago
We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on public roads, you need to convince other, possibly skeptical, people that your technology works better than humans. How do you do that with AI/ML-based systems?
- m0zg 8y ago
  >> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happen.
  - oldgradstudent 8y ago
    Imagine for a moment that you have a working self-driving car in the field, and now you want to make a relatively minor change to the system. How do you convince yourself or a regulator, not that the technology in general is safe, but that your specific change does not make it unsafe? What happens if this specific change increases the fatality rate from 1 in 100 million miles to 1 in 50 million miles? Crashes will still be rare and there's enough statistical noise that when you finally understand the change is bad, you may have killed hundreds.
  - TeMPOraL 8y ago
    > Same as with humans: from statistics. I don't see any other way. I.e. from body count. One problem here is that we know humans have common, stable brain architecture, so the limits and failure modes we experienced are stable too, and can be accounted for and worked around. People won't fail you in completely surprising ways. DNNs are each a different breed; between changing architectures and tuning hyperparameters, I don't see how trust in one instance can be translated to trust un another.