5 ms·

There was a discussion on HN last week that I can't find where I was enlightened to find out that one of the big problems around doing facial recognition is pro

by hpoe 6y ago

There was a discussion on HN last week that I can't find where I was enlightened to find out that one of the big problems around doing facial recognition is proper lighting and without it you can't really build good models and be able to really use the image. As an extension of this photographs of individuals with darker skin required more lighting than photographs of individuals with darker skin. I don't know all of what goes into the ML for facial recognition and I am sure there are people far smarter than me working on it (and making way more money than me to boot), but I guess my thought here is that some variation of Poe's Law applies. I know that people are quick to jump to condemn something as racist but sometimes there really are just honest mistakes. I have a hard time believing that anyone at any level of the AWS structure set out to produce a racist facial recognition, but rather it may have just been an honest over site and rather than rushing to crucify them we should instead look at it as a learning opportunity to help develop further the field of facial recognition. EDIT: I wanted to clarify that although I don't think it was done purposefully I don't think it doesn't bespeak a problem; my intention was rather to serve as a suggestion that we should sometimes temper the often strong reaction produced when labeling something racist, and focus our efforts on identifying and solving the issue rather than trying to act punitive. To forestall objections I do recognize this is an issue that does require correction, and that it does bespeak a larger societal problem that has real consequences for real people ever day, but in my experience we will get more progress by attempting to work together in a spirit of cooperation rather than a spirit of anger and vengance.

mc32 6y ago
I think the main thing is that even if the technology were improved and it could be proven that the biases were so low as to be negligible or that it basically identifies people equally well, the opposition would be on the technology as a tool. People want some privacy in public. A tech that can track or backtrack people's movements is kinda creepy in a few ways.
akiselev 6y ago
The technological implementation is almost certainly an honest mistake but the marketing and sales around it is not. We've known about the importance of lighting in photography for more than a century and the computer vision industry has been battling the issue for decades in manufacturing because metal is shiny. Kodak even used to manufacture different films to better represent non-white people and camera sensor manufacturers today have entire teams with a swarm of diverse models dedicated to accurate color reproduction of skin tones across races. It's been an ongoing issue for people of color ever since TV became a thing because those technologies were initially very poor at color reproduction as well, but that didn't stop newscasters from jumping face first into televised manhunts. We've been around this block several times before and while the quarry may change from photo accuracy to ML driven facial recognition, the hunt does not. There's no excuse for selling technology to industries facing (or creating) life and death situations when the bugs are so obvious.
coderintherye 6y ago
This is EXACTLY what people are trying to raise awareness of. This is implicit bias. It doesn't matter if people have "good intentions" or made "honest mistakes" if the tool is implicitly biased. This is why having a diverse team working on a project is important, because it will help call out these issues early rather than after they have been put in production. It's also not purely a technical problem, if you feed the model only pictures of white and asian male college students then it's no surprise when you get a model that biases towards recognizing white and asian male college students (which is exactly how several prominent models were trained).
- zerocrates 6y ago
  Biased algorithms/models are particularly dangerous because they tend to provide a veneer of objectivity (plausible deniability if you want to put a more cynical lens on it) that could frustrate attempts to hold users accountable.
  - dkn775 6y ago
    Agreed. A huge problem I haven't been able to think of is the (already happening in masse) practice of targeting black people because the data says so. However over time police Dept invests data capture resources in "bad areas" - which may have a lot of black residents. I wonder how to separate the defense that some black areas are obviously bad and why would a dept put surveillance resources in a good neighborhood? I personally feel it's wrong but that's one thing I've always got hung up on in building a critique.
- cpeterso 6y ago
  Mathematician Cathy O'Neil's book "Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" is a good introduction to the implicit biases in machine learning: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
  - koheripbal 6y ago
outworlder 6y ago
> As an extension of this photographs of individuals with darker skin required more lighting than photographs of individuals with darker skin. This is plain physics, no? Things are darker or lighter depending on the amount of light they reflect. > I know that people are quick to jump to condemn something as racist but sometimes there really are just honest mistakes. There are far too many things at play here. In a fair and just society, this kind of issue (like the XBOX Kinetic issue) would be met with a "oops, forgot to account for individual light absorption variance". A fix would have been issue and that's that. Now, the problem starts when you begin digging. Why was this problem not caught? Because QA teams didn't catch it. Why didn't they catch it? Because the team wasn't very diverse, so testing failed to catch the problem? Why wasn't the team diverse enough? ... and now I've entered a societal rabbit hole that's far too complex for this post. > I have a hard time believing that anyone at any level of the AWS structure set out to produce a racist facial recognition Yes, highly doubtful. No benefit, and major issues if caught. It is far more likely that the dataset itself was biased. Why was it biased? ... and there we go again. > we should sometimes temper the often strong reaction produced when labeling something racist, and focus our efforts on identifying and solving the issue rather than trying to act punitive. Agreed, in principle. In practice, this produces no results in a mostly racist society. Companies (and politicians) will listen to outrage, they won't listen to well articulated and well-reasoned comments.
- SilasX 6y ago
  >> As an extension of this photographs of individuals with darker skin required more lighting than photographs of individuals with darker skin. >This is plain physics, no? Things are darker or lighter depending on the amount of light they reflect. Not quite. I don't know all the technical terms, but apparently, photograph technology early on settled on some standards and made some design choices that made it easier to photograph white than black people. That set a long-term precedent and standard for how film should work that persisted for a while and even to the present. >>>Until recently, due to a light-skin bias embedded in colour film stock emulsions and digital camera design, the rendering of non-Caucasian skin tones was highly deficient and required the development of compensatory practices and technology improvements to redress its shortcomings. Using the emblematic “Shirley” norm reference card as a central metaphor reflecting the changing state of race relations/aesthetics, this essay analytically traces the colour adjustment processes in the industries of visual representation and identifies some prototypical changes in the field... https://www.cjc-online.ca/index.php/journal/article/view/2196 https://www.cjc-online.ca/index.php/journal/article/view/219... Lay article on the topic: https://www.buzzfeednews.com/article/syreetamcfadden/teaching-the-camera-to-see-my-skin https://www.buzzfeednews.com/article/syreetamcfadden/teachin...
btown 6y ago
If you lead a team building a facial recognition project, and you either elect not to run/look at experiments on skin color, or you ignore your findings on skin color, and decide to sell that product to people with the express and upfront intention of using it to target the use of government-sanctioned force against individuals, there's no such thing as an "honest oversight." You have actively made a decision to endanger lives. Hanlon's Razor (which I think you mean rather than Poe's Law) doesn't apply when you know you're giving violent people a force multiplier.

by hpoe 6y ago

mc32 6y ago

I think the main thing is that even if the technology were improved and it could be proven that the biases were so low as to be negligible or that it basically identifies people equally well, the opposition would be on the technology as a tool. People want some privacy in public. A tech that can track or backtrack people's movements is kinda creepy in a few ways.

akiselev 6y ago

The technological implementation is almost certainly an honest mistake but the marketing and sales around it is not. We've known about the importance of lighting in photography for more than a century and the computer vision industry has been battling the issue for decades in manufacturing because metal is shiny. Kodak even used to manufacture different films to better represent non-white people and camera sensor manufacturers today have entire teams with a swarm of diverse models dedicated to accurate color reproduction of skin tones across races. It's been an ongoing issue for people of color ever since TV became a thing because those technologies were initially very poor at color reproduction as well, but that didn't stop newscasters from jumping face first into televised manhunts. We've been around this block several times before and while the quarry may change from photo accuracy to ML driven facial recognition, the hunt does not. There's no excuse for selling technology to industries facing (or creating) life and death situations when the bugs are so obvious.

coderintherye 6y ago

This is EXACTLY what people are trying to raise awareness of. This is implicit bias. It doesn't matter if people have "good intentions" or made "honest mistakes" if the tool is implicitly biased. This is why having a diverse team working on a project is important, because it will help call out these issues early rather than after they have been put in production. It's also not purely a technical problem, if you feed the model only pictures of white and asian male college students then it's no surprise when you get a model that biases towards recognizing white and asian male college students (which is exactly how several prominent models were trained).

zerocrates 6y ago
Biased algorithms/models are particularly dangerous because they tend to provide a veneer of objectivity (plausible deniability if you want to put a more cynical lens on it) that could frustrate attempts to hold users accountable.
- dkn775 6y ago
  Agreed. A huge problem I haven't been able to think of is the (already happening in masse) practice of targeting black people because the data says so. However over time police Dept invests data capture resources in "bad areas" - which may have a lot of black residents. I wonder how to separate the defense that some black areas are obviously bad and why would a dept put surveillance resources in a good neighborhood? I personally feel it's wrong but that's one thing I've always got hung up on in building a critique.
cpeterso 6y ago
Mathematician Cathy O'Neil's book "Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" is a good introduction to the implicit biases in machine learning: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
- koheripbal 6y ago

outworlder 6y ago

> As an extension of this photographs of individuals with darker skin required more lighting than photographs of individuals with darker skin. This is plain physics, no? Things are darker or lighter depending on the amount of light they reflect. > I know that people are quick to jump to condemn something as racist but sometimes there really are just honest mistakes. There are far too many things at play here. In a fair and just society, this kind of issue (like the XBOX Kinetic issue) would be met with a "oops, forgot to account for individual light absorption variance". A fix would have been issue and that's that. Now, the problem starts when you begin digging. Why was this problem not caught? Because QA teams didn't catch it. Why didn't they catch it? Because the team wasn't very diverse, so testing failed to catch the problem? Why wasn't the team diverse enough? ... and now I've entered a societal rabbit hole that's far too complex for this post. > I have a hard time believing that anyone at any level of the AWS structure set out to produce a racist facial recognition Yes, highly doubtful. No benefit, and major issues if caught. It is far more likely that the dataset itself was biased. Why was it biased? ... and there we go again. > we should sometimes temper the often strong reaction produced when labeling something racist, and focus our efforts on identifying and solving the issue rather than trying to act punitive. Agreed, in principle. In practice, this produces no results in a mostly racist society. Companies (and politicians) will listen to outrage, they won't listen to well articulated and well-reasoned comments.

SilasX 6y ago
>> As an extension of this photographs of individuals with darker skin required more lighting than photographs of individuals with darker skin. >This is plain physics, no? Things are darker or lighter depending on the amount of light they reflect. Not quite. I don't know all the technical terms, but apparently, photograph technology early on settled on some standards and made some design choices that made it easier to photograph white than black people. That set a long-term precedent and standard for how film should work that persisted for a while and even to the present. >>>Until recently, due to a light-skin bias embedded in colour film stock emulsions and digital camera design, the rendering of non-Caucasian skin tones was highly deficient and required the development of compensatory practices and technology improvements to redress its shortcomings. Using the emblematic “Shirley” norm reference card as a central metaphor reflecting the changing state of race relations/aesthetics, this essay analytically traces the colour adjustment processes in the industries of visual representation and identifies some prototypical changes in the field... https://www.cjc-online.ca/index.php/journal/article/view/2196 https://www.cjc-online.ca/index.php/journal/article/view/219... Lay article on the topic: https://www.buzzfeednews.com/article/syreetamcfadden/teaching-the-camera-to-see-my-skin https://www.buzzfeednews.com/article/syreetamcfadden/teachin...

btown 6y ago

If you lead a team building a facial recognition project, and you either elect not to run/look at experiments on skin color, or you ignore your findings on skin color, and decide to sell that product to people with the express and upfront intention of using it to target the use of government-sanctioned force against individuals, there's no such thing as an "honest oversight." You have actively made a decision to endanger lives. Hanlon's Razor (which I think you mean rather than Poe's Law) doesn't apply when you know you're giving violent people a force multiplier.