4 ms·

That's all very cool. But for safety critical systems you have to understand how these systems work to understand their limitations. You have to know when thes

That's all very cool. But for safety critical systems you have to understand how these systems work to understand their limitations. You have to know when these techniques succeed, when they fail, and how badly they fail.

m0zg 8y ago
Do you feel like you understand human limitations in these critical systems? Are humans suitable? Would AI be suitable if it performs statistically better than humans?
- oldgradstudent 8y ago
  > Would AI be suitable if it performs statistically better than humans? In general yes, but it might depend on the pattern of failure - if your self-driving cars hunts me or my family personally, I might have problem with that. But how can you determine that without releasing it to the wild and waiting for bodies? Worse, say you have a safe system, but you need to modify the network (to fix some bug). How can you determine that the new system is safe enough to put on the road?
  - m0zg 8y ago
    But any technology can be deadly if you deploy it widely enough. _WhatsApp_ has resulted in "bodies" and it doesn't have any AI in it at all. First airplanes were basically flying coffins. Cars until early 90s had very little chance of survival in collision above 40mph. Many drugs have serious, sometimes deadly side effects. Quarter of a million people die in hospitals in US alone every year due to medical errors. 100% of those errors are currently made by humans. It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies.
    - perl4ever 8y ago
      "Quarter of a million people die in hospitals in US alone every year due to medical errors. 100% of those errors are currently made by humans." Well, that's an odd thing to say. I guess you can say that, for instance, the Therac-25 episodes involved human error, but in that case, any problems with AI are also due to human error. If everything is due to human error, then there is no alternative to defend. Medical errors certainly do involve the interface between humans and computers, and it isn't really plausible that either humans or computers can be eliminated from medicine.
    - gshrikant 8y ago
      Rightfully so. Part of the problem is the way that we speak about the technology - calling it 'artificial intelligence' when the underlying technology does not resemble true intelligence at all. This raises expectations and lets people use AI as a dumping ground for infeasible ideas.
      - m0zg 8y ago
        But my point is, even the "real" intelligence isn't so hot in a lot of cases, and AI could surpass it in terms of outcomes on narrow tasks.
    - bumby 8y ago
      My guess is that most people feel AI should be held to a higher standard is because we feel the need to be able to audit the system in the case of mishaps. When ML becomes a high-level black box, we may not have the confidence in how to right that ship if it goes astray. With human errors, if we're (hopefully) empathetic creatures we at least have the hope of understanding the root of the error.
      - RA_Fisher 8y ago
        The good news is that these tools exist but they're called statistics.
        bumby 8y ago
        Statistics applied to black-box component mishaps have a couple things going against them. 1) you need a relatively large number of failures to build a good sample of data and 2) even if you have the probabilities in place to quantify risk, you may never understand the root cause of the failure to fix or mitigate it. For a large expensive system, the presence of either of the above may be unacceptable. Take something like the space shuttle program. If it was heavily reliant on black box AI, you might be able to build probabilities through tools like monte carlo simulations but you would be hard pressed for the government to put billions of dollars at risk without understanding the root cause of simulation failures
    - oldgradstudent 8y ago
      > It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require trust. One way of establishing trust is to explain or prove why the system works. This cannot be done with machine learning. Say that you already have a safe system. Now you make a change. How do you demonstrate to a skeptical audience that the new system is safe without releasing it and counting the bodies?
      - m0zg 8y ago
        But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a story just the other day where doctors were injecting people with some proteins linked to Alzheimer's? They weren't required to be "proven that they're safe" before they did that.
        oldgradstudent 8y ago
        We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on public roads, you need to convince other, possibly skeptical, people that your technology works better than humans. How do you do that with AI/ML-based systems?
        m0zg 8y ago
        >> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happen.
  - perl4ever 8y ago
    I think people obfuscate (in general, in these discussions) the difference between "known unknowns" and "unknown unknowns". People get scared of the unquantifiable potential errors and other people pooh-pooh the idea we can't deal with risk. But is it reasonable to consider two different kinds of risk?
    - bumby 8y ago
      The issue with "unknown unknowns" is that they are uncertainties and not risks. Risks we can more easily deal with because they are quantifiable so they can be hedged against with probabilities.

m0zg 8y ago

Do you feel like you understand human limitations in these critical systems? Are humans suitable? Would AI be suitable if it performs statistically better than humans?

oldgradstudent 8y ago
> Would AI be suitable if it performs statistically better than humans? In general yes, but it might depend on the pattern of failure - if your self-driving cars hunts me or my family personally, I might have problem with that. But how can you determine that without releasing it to the wild and waiting for bodies? Worse, say you have a safe system, but you need to modify the network (to fix some bug). How can you determine that the new system is safe enough to put on the road?
- m0zg 8y ago
  But any technology can be deadly if you deploy it widely enough. _WhatsApp_ has resulted in "bodies" and it doesn't have any AI in it at all. First airplanes were basically flying coffins. Cars until early 90s had very little chance of survival in collision above 40mph. Many drugs have serious, sometimes deadly side effects. Quarter of a million people die in hospitals in US alone every year due to medical errors. 100% of those errors are currently made by humans. It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies.
  - perl4ever 8y ago
    "Quarter of a million people die in hospitals in US alone every year due to medical errors. 100% of those errors are currently made by humans." Well, that's an odd thing to say. I guess you can say that, for instance, the Therac-25 episodes involved human error, but in that case, any problems with AI are also due to human error. If everything is due to human error, then there is no alternative to defend. Medical errors certainly do involve the interface between humans and computers, and it isn't really plausible that either humans or computers can be eliminated from medicine.
  - gshrikant 8y ago
    Rightfully so. Part of the problem is the way that we speak about the technology - calling it 'artificial intelligence' when the underlying technology does not resemble true intelligence at all. This raises expectations and lets people use AI as a dumping ground for infeasible ideas.
    - m0zg 8y ago
      But my point is, even the "real" intelligence isn't so hot in a lot of cases, and AI could surpass it in terms of outcomes on narrow tasks.
  - bumby 8y ago
    My guess is that most people feel AI should be held to a higher standard is because we feel the need to be able to audit the system in the case of mishaps. When ML becomes a high-level black box, we may not have the confidence in how to right that ship if it goes astray. With human errors, if we're (hopefully) empathetic creatures we at least have the hope of understanding the root of the error.
    - RA_Fisher 8y ago
      The good news is that these tools exist but they're called statistics.
      - bumby 8y ago
        Statistics applied to black-box component mishaps have a couple things going against them. 1) you need a relatively large number of failures to build a good sample of data and 2) even if you have the probabilities in place to quantify risk, you may never understand the root cause of the failure to fix or mitigate it. For a large expensive system, the presence of either of the above may be unacceptable. Take something like the space shuttle program. If it was heavily reliant on black box AI, you might be able to build probabilities through tools like monte carlo simulations but you would be hard pressed for the government to put billions of dollars at risk without understanding the root cause of simulation failures
  - oldgradstudent 8y ago
    > It's remarkable that AI seems to be held to an arbitrarily high standard, often exceeding that of other technologies. Because safety critical systems require trust. One way of establishing trust is to explain or prove why the system works. This cannot be done with machine learning. Say that you already have a safe system. Now you make a change. How do you demonstrate to a skeptical audience that the new system is safe without releasing it and counting the bodies?
    - m0zg 8y ago
      But again, my point is, you can't explain why a human works. Nor can you predict how the human will perform in an extreme situation they haven't been through before. To modify your example. Say you have a doctor who haven't killed anyone yet and she reads a paper describing a new high risk / high reward treatment. The same standard of proof is not applied to her, so she can just go ahead and try it. Wasn't there a story just the other day where doctors were injecting people with some proteins linked to Alzheimer's? They weren't required to be "proven that they're safe" before they did that.
      - oldgradstudent 8y ago
        We can't explain how and why humans work, but we have a lot of experience with them. We know their failure modes pretty well. As to medical treatments, they are done on volunteers under informed consent. If you want to experiment with a car driven by AI on a private track staffed with volunteers, be my guest. It's between you, your conscience, your deity, and maybe OSHA. But for experimentation and deployment on public roads, you need to convince other, possibly skeptical, people that your technology works better than humans. How do you do that with AI/ML-based systems?
        m0zg 8y ago
        >> How do you do that with AI/ML-based systems? Same as with humans: from statistics. I don't see any other way. E.g. autonomous cars will never have zero accident rate, but if they have half as many accidents per million miles as humans, it's a no brainer to me that they should be deployed. We're not going to get there in the next 20 years, so I'm not saying they should be deployed _now_, but eventually it will happen.
- perl4ever 8y ago
  I think people obfuscate (in general, in these discussions) the difference between "known unknowns" and "unknown unknowns". People get scared of the unquantifiable potential errors and other people pooh-pooh the idea we can't deal with risk. But is it reasonable to consider two different kinds of risk?
  - bumby 8y ago
    The issue with "unknown unknowns" is that they are uncertainties and not risks. Risks we can more easily deal with because they are quantifiable so they can be hedged against with probabilities.