6 ms·

I don't see a problem with not being able to explain how something works as long as it's not failing empirically. Scientists also cannot explain how humans work

by kriro 4y ago

I don't see a problem with not being able to explain how something works as long as it's not failing empirically. Scientists also cannot explain how humans work in detail and yet humans are "allowed" to do many things and make many decisions that cannot be fully explained. The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.

omginternets 4y ago
There's the obvious problem of knowing if and when it might suddenly start failing empirically.
- science4sail 4y ago
  Isn't that also a problem with humans? A human pilot might suddenly go insane and start murdering people (even though empirically, it's an improbable event). https://en.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.wikipedia.org/wiki/Germanwings_Flight_9525
  - omginternets 4y ago
    If your point is just that machines can be more reliable than humans, that's not a terribly interesting argument. Humans are often held accountable for the behavior of the machines they oversee. This is no exception.
Verdex 4y ago
I'm mostly in agreement.[1] The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine. We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality. And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind. [1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.) However, even so, I would not allow such a doctor to roam free. Perhaps irrationally. The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy. For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.
- djmips 4y ago
  Not everyone is adept at storytelling I'm afraid.
rwmj 4y ago
Eventually it'll come up in a court case or similar scenario: Your AI killed a particular patient (never mind that it saved a thousand others). Can you explain to a judge why that patient was killed? Can you ensure that the changes you have made since mean it will not kill another patient? Those are going to be "interesting" questions to have to deal with in a legal or regulatory environment.
dinosaurdynasty 4y ago
One of the problems of highly capable AI ("outperforms humans empirically" in enough tasks) versus most physical systems is that it can plan and it can lie. One situation AI safety researches worry about is "AI wants X; human wants Y; AI knows human wants Y; AI does Y in training/experiments; once AI is in place to do X instead without human being able to stop AI AI does X"
- loyalcinnamon 4y ago
  Hmm, well, instead of an ai we put a human, they could do that exact thing too. So what's stoping a human? Punishment? Does this mean we need to make the ai understand punishment and/or consequences?
  - dinosaurdynasty 4y ago
    How many kids, from punishment, learn the lesson "don't do X while parents are watching" instead of "don't do X for moral reasons/etc"? How many children cut strict parents from their lives once they are self-sufficient? If you could always guarantee punishment and/or consequences this wouldn't be such a big deal in the first place.
notahacker 4y ago
> The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? Because, for a start, if you don't have a clue how it works, it's pretty difficult to have any confidence that it will actually outperform humans empirically in the long run, or in a specific set of circumstances.

by kriro 4y ago

omginternets 4y ago

There's the obvious problem of knowing if and when it might suddenly start failing empirically.

science4sail 4y ago
Isn't that also a problem with humans? A human pilot might suddenly go insane and start murdering people (even though empirically, it's an improbable event). https://en.wikipedia.org/wiki/Germanwings_Flight_9525 https://en.wikipedia.org/wiki/Germanwings_Flight_9525
- omginternets 4y ago
  If your point is just that machines can be more reliable than humans, that's not a terribly interesting argument. Humans are often held accountable for the behavior of the machines they oversee. This is no exception.

Verdex 4y ago

I'm mostly in agreement.[1] The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine. We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality. And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind. [1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.) However, even so, I would not allow such a doctor to roam free. Perhaps irrationally. The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy. For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.

djmips 4y ago
Not everyone is adept at storytelling I'm afraid.

rwmj 4y ago

Eventually it'll come up in a court case or similar scenario: Your AI killed a particular patient (never mind that it saved a thousand others). Can you explain to a judge why that patient was killed? Can you ensure that the changes you have made since mean it will not kill another patient? Those are going to be "interesting" questions to have to deal with in a legal or regulatory environment.

dinosaurdynasty 4y ago

One of the problems of highly capable AI ("outperforms humans empirically" in enough tasks) versus most physical systems is that it can plan and it can lie. One situation AI safety researches worry about is "AI wants X; human wants Y; AI knows human wants Y; AI does Y in training/experiments; once AI is in place to do X instead without human being able to stop AI AI does X"

loyalcinnamon 4y ago
Hmm, well, instead of an ai we put a human, they could do that exact thing too. So what's stoping a human? Punishment? Does this mean we need to make the ai understand punishment and/or consequences?
- dinosaurdynasty 4y ago
  How many kids, from punishment, learn the lesson "don't do X while parents are watching" instead of "don't do X for moral reasons/etc"? How many children cut strict parents from their lives once they are self-sufficient? If you could always guarantee punishment and/or consequences this wouldn't be such a big deal in the first place.

notahacker 4y ago

> The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? Because, for a start, if you don't have a clue how it works, it's pretty difficult to have any confidence that it will actually outperform humans empirically in the long run, or in a specific set of circumstances.