5 ms·
There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person decidin
by mseebach 9y ago
There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place.
Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.
- throwaway2048 9y agoIf a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
- mseebach 9y agoYes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.
- throwaway2048 9y agobut you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up
- twblalock 9y agoWhat if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.
- mseebach 9y agoIt's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.
- jwatte 9y agoWhat if a drunk drives the car into a pedestrian? Not much content there, either.
- deleted 9y ago[deleted]
- JumpCrisscross 9y ago> What if a drunk drives the car into a pedestrian? This is easy to explain. The driver is at fault. Explainability is what we are debating.
- megy 9y agoSo the AI is at fault then.
- throwaway2048 9y agothe question becomes what can you >do< about it if you have no understanding of the logic it operates on.
- TheOtherHobbes 9y agoVery much not. Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence. The software has no more agency than a cruise control.
- twblalock 9y agoGood luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.
- jwatte 9y agoThere are tons of physical processes that aren't explainable. Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? Modeling the failure, and engineering a solution that no longer fails in that situation. Unit testing our way to robustness!
- JumpCrisscross 9y ago> Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? When these processes kill people we very much do somethint about them.
- afarrell 9y ago> Becauae there’s nothing to explain. The software is broken and needs to be fixed The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.
- deleted 9y ago[deleted]
- ubernostrum 9y agoI think it could be useful to distinguish systems which are required to automatically explain every decision they recommend as a matter of routine, versus systems which are not, but still are required to support post-mortem investigation of serious failures.