7 ms·

The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being l

The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being liable or responsible, and don’t have time to verify. Plus “no one gets fired for doing what algorithm says”.

patrakov 4y ago
I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.
- auggierose 4y ago
  That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
  - krisoft 4y ago
    > Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up. Did you know that they do this with airport scanners?[1] It is very hard to maintain concentration while you scan innocent luggage after innocent luggage. And the chances of encountering a bomb or smuggled weapon is very low, most scanners will work a lifetime without ever seeing one for real. So how can we know that they concentrate instead of just blindly pushing the “ok” button? The system is programmed to preset fake weapons overimposed on the scans a certain percent of time. They are expected to push the “suspicious” button, and then the computer says “you got it, it was me who put the weapon’s image there” and asks the scanning person to check again now with the removed overlay. It is much easier to filter out one out of ten on average than one out of a million. How hellish this is depends on the communication and consequences. If your boss does this very rarely, doesn’t tell you about it, and punishes you severly that sound hellish. If they tell you that on average every 5th case you inspect will be innocent, and your task is to filter these out, then it is much harder to get worked up about it. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/
    - auggierose 4y ago
      No, I didn't know that. Yes, I agree, if the communication is upfront and clear, it seems better. But these things need to be handled with care, and the whole Dutch fiasco seems to be produced by people who do not care about the right things in the first place. Not sure also if airport security is a shining example of how much sense these methods make.
  - patrakov 4y ago
    I never said "giving wrong facts", I only meant presenting a wrong conclusion, to be nullified by human inspection of the still-correct facts.
tehwebguy 4y ago
I mean it’s not an accident. If the algorithm spit out some set of actions that they didn’t want they would just do something else or change the algorithm. The word “algorithm” has become the qualified immunity of non-police world. Seems you can always blame the algorithm and walk away unscathed.
zo1 4y ago
As someone one the receiving end of requirements to implement such algorithmic rules we were also powerless. Sure we can question it, get forwarded some random official government document and voila it gets put into a spec document. Best we could do was make it configurable on a database so it could be changed and noted that to them "in case". But at the end of the day, we essentially had a rule that said: If birth country in (Afghanistan, Serbia, Russia, ...) then add +20 risk points for flagging.
- mkmk3 4y ago
  I get you, just following orders, I earnestly empathize. I wonder how powerless software people really are in this context, how much effective oversight others have over us in these matters, where there's sometimes a significant human cost. Just a thought.

patrakov 4y ago

I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.

auggierose 4y ago
That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
- krisoft 4y ago
  > Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up. Did you know that they do this with airport scanners?[1] It is very hard to maintain concentration while you scan innocent luggage after innocent luggage. And the chances of encountering a bomb or smuggled weapon is very low, most scanners will work a lifetime without ever seeing one for real. So how can we know that they concentrate instead of just blindly pushing the “ok” button? The system is programmed to preset fake weapons overimposed on the scans a certain percent of time. They are expected to push the “suspicious” button, and then the computer says “you got it, it was me who put the weapon’s image there” and asks the scanning person to check again now with the removed overlay. It is much easier to filter out one out of ten on average than one out of a million. How hellish this is depends on the communication and consequences. If your boss does this very rarely, doesn’t tell you about it, and punishes you severly that sound hellish. If they tell you that on average every 5th case you inspect will be innocent, and your task is to filter these out, then it is much harder to get worked up about it. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8952858/
  - auggierose 4y ago
    No, I didn't know that. Yes, I agree, if the communication is upfront and clear, it seems better. But these things need to be handled with care, and the whole Dutch fiasco seems to be produced by people who do not care about the right things in the first place. Not sure also if airport security is a shining example of how much sense these methods make.
- patrakov 4y ago
  I never said "giving wrong facts", I only meant presenting a wrong conclusion, to be nullified by human inspection of the still-correct facts.

tehwebguy 4y ago

I mean it’s not an accident. If the algorithm spit out some set of actions that they didn’t want they would just do something else or change the algorithm. The word “algorithm” has become the qualified immunity of non-police world. Seems you can always blame the algorithm and walk away unscathed.

zo1 4y ago

As someone one the receiving end of requirements to implement such algorithmic rules we were also powerless. Sure we can question it, get forwarded some random official government document and voila it gets put into a spec document. Best we could do was make it configurable on a database so it could be changed and noted that to them "in case". But at the end of the day, we essentially had a rule that said: If birth country in (Afghanistan, Serbia, Russia, ...) then add +20 risk points for flagging.

mkmk3 4y ago
I get you, just following orders, I earnestly empathize. I wonder how powerless software people really are in this context, how much effective oversight others have over us in these matters, where there's sometimes a significant human cost. Just a thought.