7 ms·

52% is little better than a coin flip. If you have a million individuals in your city, your confidence should be in the ballpark of 99.9999% (1 individual in 1

52% is little better than a coin flip. If you have a million individuals in your city, your confidence should be in the ballpark of 99.9999% (1 individual in 1 million). That has really been my concern with this, the software will report any facial match above 75% confidence. Apart from the fact that it appalling confidence, no cop will pay attention to that percentage; immediately arresting or killing the individual. Software can kill. This software can kill 50% of black people.

ssss11 6y ago
What a great comment. This encapsulates my concerns about the topic eloquently. The technology is not ready for use.
cortesoft 6y ago
It would have to be even higher than that level of accuracy, because every person is going to be 'tested' multiple times.... if everyone's face is scanned 100 times a day, the number of false positives is going to be even higher.
- bigfudge 6y ago
  We shouldn’t assume those tests and errors are independent, they probably aren’t, but you are right that the overall error rate would be inflated.
dtwest 6y ago
Software can kill if we put blind trust in it and give it full control over the situation. But we shouldn't do that. Even if it was correct 99% of the time, we need to recognize that software can make mistakes. It is a tool, and people need to be responsible enough to use it correctly. I think I agree with your general idea here, but to put all of the blame on software strikes me as an incomplete assessment. Technically the software isn't killing anyone, irresponsible users of it are.
- ArtDev 6y ago
  Any software developer will tell how the marketing and sales departments will say or spin ANYTHING they can get away with to sell the product.
- toofy 6y ago
  > Technically the software isn't killing anyone, irresponsible users of it are. Sure, but at this point, we know how irresponsible users often are, we know this to be a an absolute fact. If the fact of user’s irresponsibility isn’t the centerpiece of our conversations, then we’re being incredibly irresponsible ourselves. The material manifestations of how these tools will be used has to remain at the center if researchers place any value whatsoever on our ethical responsibilities.
  - jesstaa 6y ago
    Yep, There are so many psychology studies that show groupthink , people using statements of an authority as a way remove individual responsibility and people overriding their own perceptions to agree with an authority. "I guess the computer got it wrong" is a terrifying thing for a police officer to say.
  - dzhiurgis 6y ago
    Software flies rockets, planes, ships, cars, factories and just about everything else. Yet somehow LE shouldn't be using it because... they are dumb? Everyone else is smart tho.
    - toofy 6y ago
      Did you respond to the wrong comment? I don’t believe I implied anything close to what you just said.
    - zuminator 6y ago
      If you fly a plane, drive a car or operate a factory, your livelihood and often your life depends on your constantly paying attention to the output of the software and making constant course-correcting adjustments if necessary. And the software itself often has the ability to avoid fatal errors built in. You rely on it in a narrow domain because it is highly reliable within that domain. For example, your vehicle's cruise control will generally not suddenly brake and swerve off the road so you can relax your levels of concentration to some extent. If it were only 52% likely to be maintaining your velocity and heading from moment to moment, you wouldn't trust it for a second. Facial recognition software doesn't have the level of reliability that control software for mechanical systems has. And if a mistake is made, the consequences to the LEO have been historically minimal. Shoot first and ask questions later has been deemed acceptable conduct, so why not implicitly trust in the software? If it's right and you kill a terrorist, you're a hero. If it's wrong and you kill a civilian, the US Supreme Court has stated, "Where the officer has probable cause to believe that the suspect poses a threat of serious physical harm, either to the officer or to others, it is not constitutionally unreasonable to prevent escape by using deadly force." The software provides probable cause, the subject's life is thereby forfeit. From the perspective of the officer, seems a no-brainer.
jml7c5 6y ago
Just to be clear, parent is describing fictional software, not the system in the article. You seem to be conflating the two.
- TheSpiceIsLife 6y ago
  Amusing. At this point facial recognition is fictional software.
loup-vaillant 6y ago
I would hope that the software/ML engineers who wrote it knows about probability theory, and why the prior probability should be set at 0.0001% or so. So that if we print 52% on the screen, that means we've already gathered like 30-bits of evidence (30 coin flips all coming up heads), at which point the suspicion would be real.
threatofrain 6y ago
> 99.9999% Then the justice system would implode. Judicial policy is "software" too, and nobody holds the judiciary or police to that absurd level of excellence, even if we're talking about the death penalty.
- codeisawesome 6y ago
  Wow, fuck you.
- hrktb 6y ago
  > even if we're talking about the death penalty. And that's also the core argument why some countries abolished death penalty.
- filoeleven 6y ago
  The justice system would implode if half the innocent people strong-armed into taking plea deals (with threats of much harsher sentences if they go to court) chose not to take them. That “software” is already buggy AF and needs some fundamental fixes. Setting a high standard for some crazy new AI stuff is a smaller change than fixing what’s already broken.
sokoloff 6y ago
It’s not at all obvious to me that the accuracy threshold should scale with city size. Some small town shouldn’t use a system that is 1000x less accurate.
Y-Bopinator 6y ago
2021: 50% of black people are now dead due to facial recognition. Trump suspends summary execution.

ssss11 6y ago

What a great comment. This encapsulates my concerns about the topic eloquently. The technology is not ready for use.

cortesoft 6y ago

It would have to be even higher than that level of accuracy, because every person is going to be 'tested' multiple times.... if everyone's face is scanned 100 times a day, the number of false positives is going to be even higher.

bigfudge 6y ago
We shouldn’t assume those tests and errors are independent, they probably aren’t, but you are right that the overall error rate would be inflated.

dtwest 6y ago

Software can kill if we put blind trust in it and give it full control over the situation. But we shouldn't do that. Even if it was correct 99% of the time, we need to recognize that software can make mistakes. It is a tool, and people need to be responsible enough to use it correctly. I think I agree with your general idea here, but to put all of the blame on software strikes me as an incomplete assessment. Technically the software isn't killing anyone, irresponsible users of it are.

ArtDev 6y ago
Any software developer will tell how the marketing and sales departments will say or spin ANYTHING they can get away with to sell the product.
toofy 6y ago
> Technically the software isn't killing anyone, irresponsible users of it are. Sure, but at this point, we know how irresponsible users often are, we know this to be a an absolute fact. If the fact of user’s irresponsibility isn’t the centerpiece of our conversations, then we’re being incredibly irresponsible ourselves. The material manifestations of how these tools will be used has to remain at the center if researchers place any value whatsoever on our ethical responsibilities.
- jesstaa 6y ago
  Yep, There are so many psychology studies that show groupthink , people using statements of an authority as a way remove individual responsibility and people overriding their own perceptions to agree with an authority. "I guess the computer got it wrong" is a terrifying thing for a police officer to say.
- dzhiurgis 6y ago
  Software flies rockets, planes, ships, cars, factories and just about everything else. Yet somehow LE shouldn't be using it because... they are dumb? Everyone else is smart tho.
  - toofy 6y ago
    Did you respond to the wrong comment? I don’t believe I implied anything close to what you just said.
  - zuminator 6y ago
    If you fly a plane, drive a car or operate a factory, your livelihood and often your life depends on your constantly paying attention to the output of the software and making constant course-correcting adjustments if necessary. And the software itself often has the ability to avoid fatal errors built in. You rely on it in a narrow domain because it is highly reliable within that domain. For example, your vehicle's cruise control will generally not suddenly brake and swerve off the road so you can relax your levels of concentration to some extent. If it were only 52% likely to be maintaining your velocity and heading from moment to moment, you wouldn't trust it for a second. Facial recognition software doesn't have the level of reliability that control software for mechanical systems has. And if a mistake is made, the consequences to the LEO have been historically minimal. Shoot first and ask questions later has been deemed acceptable conduct, so why not implicitly trust in the software? If it's right and you kill a terrorist, you're a hero. If it's wrong and you kill a civilian, the US Supreme Court has stated, "Where the officer has probable cause to believe that the suspect poses a threat of serious physical harm, either to the officer or to others, it is not constitutionally unreasonable to prevent escape by using deadly force." The software provides probable cause, the subject's life is thereby forfeit. From the perspective of the officer, seems a no-brainer.

jml7c5 6y ago

Just to be clear, parent is describing fictional software, not the system in the article. You seem to be conflating the two.

TheSpiceIsLife 6y ago
Amusing. At this point facial recognition is fictional software.

loup-vaillant 6y ago

I would hope that the software/ML engineers who wrote it knows about probability theory, and why the prior probability should be set at 0.0001% or so. So that if we print 52% on the screen, that means we've already gathered like 30-bits of evidence (30 coin flips all coming up heads), at which point the suspicion would be real.

threatofrain 6y ago

> 99.9999% Then the justice system would implode. Judicial policy is "software" too, and nobody holds the judiciary or police to that absurd level of excellence, even if we're talking about the death penalty.

codeisawesome 6y ago
Wow, fuck you.
hrktb 6y ago
> even if we're talking about the death penalty. And that's also the core argument why some countries abolished death penalty.
filoeleven 6y ago
The justice system would implode if half the innocent people strong-armed into taking plea deals (with threats of much harsher sentences if they go to court) chose not to take them. That “software” is already buggy AF and needs some fundamental fixes. Setting a high standard for some crazy new AI stuff is a smaller change than fixing what’s already broken.

sokoloff 6y ago

It’s not at all obvious to me that the accuracy threshold should scale with city size. Some small town shouldn’t use a system that is 1000x less accurate.

Y-Bopinator 6y ago

2021: 50% of black people are now dead due to facial recognition. Trump suspends summary execution.