6 ms·

Even if it was no-one's intent that the system is racially biased, allowing such a system to continue operate after learning of such a bias is a racist act.

lgessler 10y ago
I'm inclined to agree with you, but what would you say if it turned out that after much effort the engineering team was unable to make performance equal across all skin tones?
- jacquesm 10y ago
  You mean that it would work better with darker skin?
  - handedness 10y ago
    What if the technical challenges of identifying a person's face mean it will always be challenging to identify someone when the light and/or contrast levels are lower? What then? Manually cripple the system and misidentify white suspects, too? This post's comments are a great example of why technologists are fairly terrible at engineering social solutions, much the same way sociologists tend to be terrible at engineering technical solutions.
handedness 10y ago
Ignoring known technical deficiencies in the system and assigning the same confidence levels in the output regardless of race could work out to be pretty racist, but what are they supposed to do, manually cripple the accuracy of the system down to the lowest common denominator? In the medical device I mentioned above, it was medically useful for caucasians and lighter-skinned Hispanics, so-so with Asians, and almost useless for Africans. So, what–don't go to market until the technology has parity across all skin tones? Not everything is about intent.
- bilbo0s 10y ago
  I hate to be a pedant here... but I actually wrote a pretty successful 3d radiology/cardiology package. So I understand medical devices. I just wanted to hop in here and say that there is NO WAY the FDA would give you clearance to use the device you describe on African Americans under the circumstances that your posts have postulated. It seems a no brainer that the facial recognition software should not be used on African Americans either. At least, if it was MEDICAL software... it would DEFINITELY NOT be cleared with that kind of a failure rate. And any provider or delivery network using the facial recognition software on African Americans would be on his or her way to prison. But the regulations on legal software may be more lackadaisical than the regulations on medical software. Getting things exactly right may not be as important. And the penalties for misuse of medical devices are, without question, far more draconian.
  - handedness 10y ago
    > I just wanted to hop in here and say that there is NO WAY the FDA would give you clearance to use the device you describe on African Americans under the circumstances that your posts have postulated. The FDA has approved drugs that are specifically intended to target medical issues in specific ethnicities, so I'm not sure about your assessment. That said, this product's market was outside the scope of the FDA, so it's irrelevant. > It seems a no brainer that the facial recognition software should not be used on African Americans either. Any data source which produces data that's better than a random guess should be considered when trying to solve a problem. That it offends some people's social sensibilities is a separate issue. Look, I don't like the mere existence of the FBI's system, and I don't think the public's continued lack of concern for placing constitutional restraints on systems like it is a wise course forward, but to say it's a "no-brainer" when it might produce valuable leads is to focus on one aspect of the problem without considering the rest of it. Just as it would also be a mistake to consider it more accurate than it is when trying to identify dark-skinned suspects. Witness accuracy is not a new problem. It's also worth noting that the human eye is just as bad at facial recognition when the light and/or contrast is lower.
    - 10y ago

lgessler 10y ago

I'm inclined to agree with you, but what would you say if it turned out that after much effort the engineering team was unable to make performance equal across all skin tones?

jacquesm 10y ago
You mean that it would work better with darker skin?
- handedness 10y ago
  What if the technical challenges of identifying a person's face mean it will always be challenging to identify someone when the light and/or contrast levels are lower? What then? Manually cripple the system and misidentify white suspects, too? This post's comments are a great example of why technologists are fairly terrible at engineering social solutions, much the same way sociologists tend to be terrible at engineering technical solutions.

handedness 10y ago

Ignoring known technical deficiencies in the system and assigning the same confidence levels in the output regardless of race could work out to be pretty racist, but what are they supposed to do, manually cripple the accuracy of the system down to the lowest common denominator? In the medical device I mentioned above, it was medically useful for caucasians and lighter-skinned Hispanics, so-so with Asians, and almost useless for Africans. So, what–don't go to market until the technology has parity across all skin tones? Not everything is about intent.

bilbo0s 10y ago
I hate to be a pedant here... but I actually wrote a pretty successful 3d radiology/cardiology package. So I understand medical devices. I just wanted to hop in here and say that there is NO WAY the FDA would give you clearance to use the device you describe on African Americans under the circumstances that your posts have postulated. It seems a no brainer that the facial recognition software should not be used on African Americans either. At least, if it was MEDICAL software... it would DEFINITELY NOT be cleared with that kind of a failure rate. And any provider or delivery network using the facial recognition software on African Americans would be on his or her way to prison. But the regulations on legal software may be more lackadaisical than the regulations on medical software. Getting things exactly right may not be as important. And the penalties for misuse of medical devices are, without question, far more draconian.
- handedness 10y ago
  > I just wanted to hop in here and say that there is NO WAY the FDA would give you clearance to use the device you describe on African Americans under the circumstances that your posts have postulated. The FDA has approved drugs that are specifically intended to target medical issues in specific ethnicities, so I'm not sure about your assessment. That said, this product's market was outside the scope of the FDA, so it's irrelevant. > It seems a no brainer that the facial recognition software should not be used on African Americans either. Any data source which produces data that's better than a random guess should be considered when trying to solve a problem. That it offends some people's social sensibilities is a separate issue. Look, I don't like the mere existence of the FBI's system, and I don't think the public's continued lack of concern for placing constitutional restraints on systems like it is a wise course forward, but to say it's a "no-brainer" when it might produce valuable leads is to focus on one aspect of the problem without considering the rest of it. Just as it would also be a mistake to consider it more accurate than it is when trying to identify dark-skinned suspects. Witness accuracy is not a new problem. It's also worth noting that the human eye is just as bad at facial recognition when the light and/or contrast is lower.
  - 10y ago