5 ms·

A common theme seems to be humans taking output from an algorithm and blindly making decisions based on it, even when specifically told that the output shouldn'

by roxgib 4y ago

A common theme seems to be humans taking output from an algorithm and blindly making decisions based on it, even when specifically told that the output shouldn't be used as the sole basis for a decision or even as evidence as all, other than cause to apply greater scrutiny. We've seen this in pharmacies flat out refusing to dispense medicine when an automated system flags a concern, even if it's simply supposed to be a prompt to ask more questions, or police trusting the results of facial recognition without further checking they have the right person. People talk about AI enhancing human abilities, and it can obviously do that, but it's really scary how willing people are to trust a system they've been told shouldn't be trusted.

kranke155 4y ago
The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being liable or responsible, and don’t have time to verify. Plus “no one gets fired for doing what algorithm says”.
- patrakov 4y ago
  I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.
  - auggierose 4y ago
    That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
    - krisoft 4y ago
      > Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up. Did you know that they do this with airport scanners?[1] It is very hard to maintain concentration while you scan innocent luggage after innocent luggage. And the chances of encountering a bomb or smuggled weapon is very low, most scanners will work a lifetime without ever seeing one for real. So how can we know that they concentrate instead of just blindly pushing the “ok” button? The system is programmed to preset fake weapons overimposed on the scans a certain percent of time. They are expected to push the “suspicious” button, and then the computer says “you got it, it was me who put the weapon’s image there” and asks the scanning person to check again now with the removed overlay. It is much easier to filter out one out of ten on average than one out of a million. How hellish this is depends on the communication and consequences. If your boss does this very rarely, doesn’t tell you about it, and punishes you severly that sound hellish. If they tell you that on average every 5th case you inspect will be innocent, and your task is to filter these out, then it is much harder to get worked up about it. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/
      - auggierose 4y ago
        No, I didn't know that. Yes, I agree, if the communication is upfront and clear, it seems better. But these things need to be handled with care, and the whole Dutch fiasco seems to be produced by people who do not care about the right things in the first place. Not sure also if airport security is a shining example of how much sense these methods make.
    - patrakov 4y ago
      I never said "giving wrong facts", I only meant presenting a wrong conclusion, to be nullified by human inspection of the still-correct facts.
- tehwebguy 4y ago
  I mean it’s not an accident. If the algorithm spit out some set of actions that they didn’t want they would just do something else or change the algorithm. The word “algorithm” has become the qualified immunity of non-police world. Seems you can always blame the algorithm and walk away unscathed.
- zo1 4y ago
  As someone one the receiving end of requirements to implement such algorithmic rules we were also powerless. Sure we can question it, get forwarded some random official government document and voila it gets put into a spec document. Best we could do was make it configurable on a database so it could be changed and noted that to them "in case". But at the end of the day, we essentially had a rule that said: If birth country in (Afghanistan, Serbia, Russia, ...) then add +20 risk points for flagging.
  - mkmk3 4y ago
    I get you, just following orders, I earnestly empathize. I wonder how powerless software people really are in this context, how much effective oversight others have over us in these matters, where there's sometimes a significant human cost. Just a thought.
dsfyu404ed 4y ago
>even when specifically told that the output shouldn't be used as the sole basis for a decision or even as evidence as all, other than cause to apply greater scrutiny It's just being used as a scapegoat to do things they don't want responsibility for. "Computer says government gets to keep all the money, instructions for an appeal we'll deny right out of the gate are on the back of your printout, have a nice day, NEXT" >but it's really scary how willing people are to trust a system they've been told shouldn't be trusted. It's not trust. It's motivated reasoning.
hotpotamus 4y ago
Have you seen "Barry"? It went over this in the context of how streaming services make decisions on their shows in a way that basically had me screaming with laughter. Essentially an executive just straight up says that after a lifetime in the business, she's learned that no one knows anything except for the algorithm, and that she's powerless to disagree with it (despite obviously having the power to disagree with it). For that and other bits, it's among the best TV currently being made.

by roxgib 4y ago

kranke155 4y ago

The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being liable or responsible, and don’t have time to verify. Plus “no one gets fired for doing what algorithm says”.

patrakov 4y ago
I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.
- auggierose 4y ago
  That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
  - krisoft 4y ago
    > Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up. Did you know that they do this with airport scanners?[1] It is very hard to maintain concentration while you scan innocent luggage after innocent luggage. And the chances of encountering a bomb or smuggled weapon is very low, most scanners will work a lifetime without ever seeing one for real. So how can we know that they concentrate instead of just blindly pushing the “ok” button? The system is programmed to preset fake weapons overimposed on the scans a certain percent of time. They are expected to push the “suspicious” button, and then the computer says “you got it, it was me who put the weapon’s image there” and asks the scanning person to check again now with the removed overlay. It is much easier to filter out one out of ten on average than one out of a million. How hellish this is depends on the communication and consequences. If your boss does this very rarely, doesn’t tell you about it, and punishes you severly that sound hellish. If they tell you that on average every 5th case you inspect will be innocent, and your task is to filter these out, then it is much harder to get worked up about it. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/
    - auggierose 4y ago
      No, I didn't know that. Yes, I agree, if the communication is upfront and clear, it seems better. But these things need to be handled with care, and the whole Dutch fiasco seems to be produced by people who do not care about the right things in the first place. Not sure also if airport security is a shining example of how much sense these methods make.
  - patrakov 4y ago
    I never said "giving wrong facts", I only meant presenting a wrong conclusion, to be nullified by human inspection of the still-correct facts.
tehwebguy 4y ago
I mean it’s not an accident. If the algorithm spit out some set of actions that they didn’t want they would just do something else or change the algorithm. The word “algorithm” has become the qualified immunity of non-police world. Seems you can always blame the algorithm and walk away unscathed.
zo1 4y ago
As someone one the receiving end of requirements to implement such algorithmic rules we were also powerless. Sure we can question it, get forwarded some random official government document and voila it gets put into a spec document. Best we could do was make it configurable on a database so it could be changed and noted that to them "in case". But at the end of the day, we essentially had a rule that said: If birth country in (Afghanistan, Serbia, Russia, ...) then add +20 risk points for flagging.
- mkmk3 4y ago
  I get you, just following orders, I earnestly empathize. I wonder how powerless software people really are in this context, how much effective oversight others have over us in these matters, where there's sometimes a significant human cost. Just a thought.

dsfyu404ed 4y ago

>even when specifically told that the output shouldn't be used as the sole basis for a decision or even as evidence as all, other than cause to apply greater scrutiny It's just being used as a scapegoat to do things they don't want responsibility for. "Computer says government gets to keep all the money, instructions for an appeal we'll deny right out of the gate are on the back of your printout, have a nice day, NEXT" >but it's really scary how willing people are to trust a system they've been told shouldn't be trusted. It's not trust. It's motivated reasoning.

hotpotamus 4y ago

Have you seen "Barry"? It went over this in the context of how streaming services make decisions on their shows in a way that basically had me screaming with laughter. Essentially an executive just straight up says that after a lifetime in the business, she's learned that no one knows anything except for the algorithm, and that she's powerless to disagree with it (despite obviously having the power to disagree with it). For that and other bits, it's among the best TV currently being made.