5 ms·

This is what social psychologists call "automation bias," which has been empirically observed in all kinds of situations from all kinds of people: https://en.wi

This is what social psychologists call "automation bias," which has been empirically observed in all kinds of situations from all kinds of people: https://en.wikipedia.org/wiki/Automation_bias https://en.wikipedia.org/wiki/Automation_bias When we are told that a prediction is from an automated system, most people are more likely to believe it and not subject it to the same skepticism and critical thinking that we do to predictions from humans. Even if there is a human-in-the-loop system that requires a human to confirm it, that's often just laundering the machine's prediction through a human. When we are primed by being told this is what The System predicted, we're more likely to override our normal processes. This is especially the case when The System has worked 99% of the time --- which is still a lot of false positives. Although this can also be the case with priming from predictions or judgments by other humans. The Asch conformity experiments involved planting fake test subjects in a group activity where subjects had to give answers to a very obvious perception question, and the real test subject could hear everyone else's answers. The fake subjects went first and all gave the blatantly wrong answer. The percent of people who conform varies with how you run the experiment, but typically a majority of the subjects end up giving the obvious wrong answer that everyone else did: https://en.wikipedia.org/wiki/Asch_conformity_experiments https://en.wikipedia.org/wiki/Asch_conformity_experiments

pbhjpbhj 6y ago
I can see it happening when checking takes a long time - but they're probably man-days into an arrest at the point they're sitting in an interview room post-arrest. Against, the LEO looking at their tablet screen prior to arresting the guy, seeing he looks different and not arresting him? Are the LEOs getting bonuses for raw number-of-arrests?
- WrtCdEvrydy 6y ago
  > Are the LEOs getting bonuses for raw number-of-arrests? There may be a subconscious bias here... if you spend your entire year looking at murder suspects, you probably see the guy on the other side as just another scumbag... and probably want to make sure the victim gets justice (even if it's wrong)
  - teddyh 6y ago
    When LEOs manage to get a conviction, they now have irrefutable proof (a conviction in a court of law) that they were right in the first place to make sure that scumbag was convicted. It’s a self-perpetuating cycle of increasing prejudice.
    - WrtCdEvrydy 6y ago
      It's real interesting going through forensics courses and working with computer forensics professionals and seeing the opposite. Since you're not briefed on the crime and only given a list of tasks to perform (ie, locate pictures of a vehicle), there might be less of a bias here.
- monocasa 6y ago
  It depends on the department, but raw arrest counts are absolutely part of individual officer performance metrics, with low arrest count officers being "pressured to find something" at the end of the month. https://www.nbcnewyork.com/news/local/nypd-officers-arrest-quota-exclusive-interview-pressure-numbers/748594/ https://www.nbcnewyork.com/news/local/nypd-officers-arrest-q...
  - BelleOfTheBall 6y ago
    Which is absolutely a big issue with our policing system. Most of the time they just arrest easy targets like sex workers or people who could be charged with 'public misconduct' or something like that.
- HarryHirsch 6y ago
  Are the LEOs getting bonuses for raw number-of-arrests? A good number are nazis, so if the software enables them to enjoy life better by harassing a few more black youth, that's enough of a reward.
  - teddyh 6y ago
    No, that’s way too easy an answer. It’s also othering the problem away; essentially saying “non-nazis would never behave this way”. But they would, have, and do. Since this is a programmer-rich forum, let’s talk about bug fixing for a moment. When you are looking for a bug, it’s easy to convince yourself a number of things, like: “It’s a random glitch, just restart it and it won’t happen again.”, or “It’s this library which is buggy, it’ll be fixed when we upgrade it to the latest patch.”. Now, of course, we programmers are then punished mightily for our arrogance when the bug keeps happening. But what would happen to bug fixing if this never happened? If we could just declare that the bug was this or that, and move on?
    - HarryHirsch 6y ago
      Obviously the police deal with crime and disorder every day, this would attract people who are more on the right of the political spectrum. This is exacerbated by deep-seated structural factors in the US, policing by consent is unknown, there is high income inequality and segregation, the level of training is pitiful, it's no surprise that Amber Guyger's instagram feed looked the way it did. Of course those types would use any tool they are handed to have fun at the people's expense. The system selects for them.
      - teddyh 6y ago
        This is still just othering police, and thereby avoiding solving any problem. I.e. if the problem is caused by some inherent factor in the people who choose to become police officers, then we can simply blame them and move on; easy hate clicks galore! But I think police are molded by their experiences, and made by various vicious cycles to become the stereotypical bad police. I described one such cycle above; here is another: http://tailsteak.com/archive.php?num=545 http://tailsteak.com/archive.php?num=545 If this is correct, then anybody, even the most well-meaning angelic people, would, over time, become bad police, but anyone could still merely claim that all police are bad because they are “right-wing” (as if that automatically is a bad thing) or racist, and nothing could ever be solved.
ivanhoe 6y ago
It's also partially the blame of companies that sell the software, as it's them that should really insist on a need for users of their systems to manually double check every automated result - but "our software makes mistakes" doesn't make for a great sales pitch. So instead of training people to doubt the results and use them only as a suggestion, they greatly exaggerate the capabilities of their software and make officers lazy about fact-checking the finds.
tgflynn 6y ago
So give your human validator a lineup - not 1 match but say the top 5 matches plus a few chosen randomly from lower down the confidence list, presented in a random order with no information on which was the algorithm's top choice.
deleted 6y ago
[deleted]

pbhjpbhj 6y ago

I can see it happening when checking takes a long time - but they're probably man-days into an arrest at the point they're sitting in an interview room post-arrest. Against, the LEO looking at their tablet screen prior to arresting the guy, seeing he looks different and not arresting him? Are the LEOs getting bonuses for raw number-of-arrests?

WrtCdEvrydy 6y ago
> Are the LEOs getting bonuses for raw number-of-arrests? There may be a subconscious bias here... if you spend your entire year looking at murder suspects, you probably see the guy on the other side as just another scumbag... and probably want to make sure the victim gets justice (even if it's wrong)
- teddyh 6y ago
  When LEOs manage to get a conviction, they now have irrefutable proof (a conviction in a court of law) that they were right in the first place to make sure that scumbag was convicted. It’s a self-perpetuating cycle of increasing prejudice.
  - WrtCdEvrydy 6y ago
    It's real interesting going through forensics courses and working with computer forensics professionals and seeing the opposite. Since you're not briefed on the crime and only given a list of tasks to perform (ie, locate pictures of a vehicle), there might be less of a bias here.
monocasa 6y ago
It depends on the department, but raw arrest counts are absolutely part of individual officer performance metrics, with low arrest count officers being "pressured to find something" at the end of the month. https://www.nbcnewyork.com/news/local/nypd-officers-arrest-quota-exclusive-interview-pressure-numbers/748594/ https://www.nbcnewyork.com/news/local/nypd-officers-arrest-q...
- BelleOfTheBall 6y ago
  Which is absolutely a big issue with our policing system. Most of the time they just arrest easy targets like sex workers or people who could be charged with 'public misconduct' or something like that.
HarryHirsch 6y ago
Are the LEOs getting bonuses for raw number-of-arrests? A good number are nazis, so if the software enables them to enjoy life better by harassing a few more black youth, that's enough of a reward.
- teddyh 6y ago
  No, that’s way too easy an answer. It’s also othering the problem away; essentially saying “non-nazis would never behave this way”. But they would, have, and do. Since this is a programmer-rich forum, let’s talk about bug fixing for a moment. When you are looking for a bug, it’s easy to convince yourself a number of things, like: “It’s a random glitch, just restart it and it won’t happen again.”, or “It’s this library which is buggy, it’ll be fixed when we upgrade it to the latest patch.”. Now, of course, we programmers are then punished mightily for our arrogance when the bug keeps happening. But what would happen to bug fixing if this never happened? If we could just declare that the bug was this or that, and move on?
  - HarryHirsch 6y ago
    Obviously the police deal with crime and disorder every day, this would attract people who are more on the right of the political spectrum. This is exacerbated by deep-seated structural factors in the US, policing by consent is unknown, there is high income inequality and segregation, the level of training is pitiful, it's no surprise that Amber Guyger's instagram feed looked the way it did. Of course those types would use any tool they are handed to have fun at the people's expense. The system selects for them.
    - teddyh 6y ago
      This is still just othering police, and thereby avoiding solving any problem. I.e. if the problem is caused by some inherent factor in the people who choose to become police officers, then we can simply blame them and move on; easy hate clicks galore! But I think police are molded by their experiences, and made by various vicious cycles to become the stereotypical bad police. I described one such cycle above; here is another: http://tailsteak.com/archive.php?num=545 http://tailsteak.com/archive.php?num=545 If this is correct, then anybody, even the most well-meaning angelic people, would, over time, become bad police, but anyone could still merely claim that all police are bad because they are “right-wing” (as if that automatically is a bad thing) or racist, and nothing could ever be solved.

ivanhoe 6y ago

It's also partially the blame of companies that sell the software, as it's them that should really insist on a need for users of their systems to manually double check every automated result - but "our software makes mistakes" doesn't make for a great sales pitch. So instead of training people to doubt the results and use them only as a suggestion, they greatly exaggerate the capabilities of their software and make officers lazy about fact-checking the finds.

tgflynn 6y ago

So give your human validator a lineup - not 1 match but say the top 5 matches plus a few chosen randomly from lower down the confidence list, presented in a random order with no information on which was the algorithm's top choice.

deleted 6y ago

[deleted]