14 ms·

I heard about different research like this about six years ago. So the algorithm wasn't tested on dark-skinned people. Is this really something new? Product dev

I heard about different research like this about six years ago. So the algorithm wasn't tested on dark-skinned people. Is this really something new? Product developers in a wide variety of markets still neglect to take into account dark skinned peoples when developing their products. Why should it come as a shock that facial recognition product developers suffer from the same bias?

Iolaum 7y ago
We should it be OK that everyone makes mistake A just because some people did it?
headalgorithm 7y ago
From the article: "Those disparities can sometimes be a matter of life or death: One recent study of the computer vision systems that enable self-driving cars to “see” the road shows they have a harder time detecting pedestrians with darker skin tones." Just think about the consequences of deploying such systems.
- js2 7y ago
  The story doesn’t cite the specific paper. But I think I submitted the paper last month: “Predictive Inequality in Object Detection” https://news.ycombinator.com/item?id=19379827 https://news.ycombinator.com/item?id=19379827 https://arxiv.org/abs/1902.11097 https://arxiv.org/abs/1902.11097 I’m not denying the conclusion of the paper, but it does have a lot of limitations. Also, Tesla notwithstanding, isn’t object detection primarily done via lidar?
- jfk13 7y ago
  I wonder if that's also true for human drivers. I certainly have a harder time detecting pedestrians wearing darker clothing, at least in some circumstances.
  - skookumchuck 7y ago
    That's why road workers, cyclists, and joggers wear brightly colored reflective vests.
  - munchbunny 7y ago
    There is also an important ethical difference between dark skin tones and dark clothing. I see the point you are making, but to the extent that an AI can do better than a human due to the physical aspect, for example, of dark skin reflecting less light at night, I think we should at least try.
- tropo 7y ago
  The computer vision systems could still beat real human vision systems. This could be by absolute numbers crashed into, or by having less bias than humans, or by both. It's not even a bias problem. It's a lighting, contrast, and camo problem. You could as well claim a bias between asphalt and concrete pavement, based on what skin tones are more visible with the pavement as a background.
  - externalreality 7y ago
    > The computer vision systems could still beat real human vision systems. For pattern recognition, No, the computer vision system can't best a human.
    - tropo 7y ago
      I'm not saying it does today. I'm saying it could do so, while still having bias. The existence of bias doesn't make the computer worse than humans.
    - deogeo 7y ago
      But the entire self-driving system can. It is never tired, or distracted, has far fewer blind spots, and possibly LIDAR and IR to help it.
      - scottie_m 7y ago
        It can’t today, and until it proves to do so making claims about future tech in relation to present policy is a bad idea.
  - int_19h 7y ago
    Suppose the system does beat real human vision by absolute numbers , but it turns out that it is objectively worse for someone who is not white, and the higher average is because the majority of people in that society are white. Is that acceptable?
    - deleted 7y ago
- colllectorof 7y ago
  So maybe we should not be using opaque ML algorithms for life-and-death situations to begin with? Instead of, you know, hoping that whomever is screwed by them has an active advocacy group in places like MIT.
  - mikeash 7y ago
    Using NL for such things hasn’t been working too great, so it seems worthwhile to try for something better.
    - hprotagonist 7y ago
      such as?
      - mikeash 7y ago
        Are you asking for examples where NL hasn’t done well or for potential ways to do better?
        hprotagonist 7y ago
        the latter. It’s relatively easy to say “stop doing X”; it’s harder to say “...and have you considered Y instead?”
        mikeash 7y ago
        I’d hope it was obvious from context that ML was the potential way to do better here.
        hprotagonist 7y ago
        I read you as saying that it was the thing doing poorly and that some superior object detection method was required. What that approach could be is totally unobvious to me.
- sonnyblarney 7y ago
  "Just think about the consequences of deploying such systems." Those systems are already widely deployed - they're called 'humans'. All vision systems are going to be more likely to confuse like colours. This problem is quite a bit different from the 'mugshot' problem, or a problem wherein there was a bias or lack of training data for certain samples. The author I think conflated a lot of the issues and just boiled it down to 'evil AI' which I don't think is the right thing to do. Issues of ethnic orientation are quite a bit different from systems that have trouble literally due to the colour of something.
- maskymask 7y ago
  I saw a talk by one of the authors of the pedestrian detection study that I think they are talking about, and their results did not seem support the scary racist self-driving car hypothesis that the study clearly implies (only marginal/statistically insignificant effects, with very little controls for confounding factors (I believe they controlled for bounding box size or something, but very little else)). With that said, fairness in ML/AI is a real problem, and some people are doing some really good/important work in this area. I am not familiar with Buolamwini's work, but I'm much more inclined to believe disparities in facial recognition than pedestrian detection where very little skin is visible and almost everything you're seeing is clothing.
- deleted 7y ago
  [deleted]
m0zg 7y ago
Edit: this factual description, written by a practitioner in the field of computer vision, of why the algorithms behave the way they do and what can be done about it, was downvoted and deliberately misinterpreted by the fake outrage mob. So I deleted it. I should have known better than to comment on a topic predisposed to attracting such people.
- m0zg 7y ago
  Edit: deleted
  - danharaj 7y ago
    > you will get better at recognizing dark skinned people, at the expense of getting worse at recognizing light skinned ones, which generally constitute the majority in the real world Sure, buddy.
    - m0zg 7y ago
      In the US that's not up for debate. That's why you omitted "in most parts of the US" part of my message, I suppose, to manufacture outrage.
      - danharaj 7y ago
        You were making a value judgment about optimal utility. That's not a fact. They're just Internet Points, comrade. > I suppose, to manufacture outrage. You're the one who seems outraged =^)
        m0zg 7y ago
        I made no judgment at all. I explained why the algorithms behave the way they do, and that there's currently no real way to make them behave better on darker skinned people without hurting accuracy overall, assuming the data distribution in the wild is not perfectly balanced, which it is not. That's literally all I said. Whatever you make up in your perfervid imagination is just that, your imagination, comrade.
mannykannot 7y ago
No-one is saying it is a shock - we're saying it is wrong and should be fixed. As for Amazon's self-serving response, it should be given the same degree of respect as any other statement by an entity that is not prepared to discuss it in an interview.
- CryptoPunk 7y ago
  Any product is going be a poorer fit for customers fitting a less common profile than those fitting a more common one. You will never get rid of that. Even if performance between races equalized due to a disproportionate amount of resources being expended on adapting the product to fit those belonging to minority racial groups, performance between unusual and common facial types or minority and majority ethnic groups within the races would exist. There will always be some minority group that the product fits less well with, because you can divide people into an infinite number of groups based on an infinite number of traits. This is a consequence of R&D resources being finite and the complexity of the world being unbounded. It's not sign of a moral failing or mis-placed priorities.
  - externalreality 7y ago
    I agree with you. However, without trying to take your points down a slippery slope, your points don't apply to safety concerns. We can't have robo-cars hitting black people because the designers neglected to test the pattern recognition libraries on dark-skinned people. That, you can probably admit, is different from a bar of soap making peoples' skin a bit dry, or lotion not taking into account melanin, or the game designer neglecting the Linux community.
    - CryptoPunk 7y ago
      The accident rate should be decreased in the most efficient way possible, to save as many lives as possible. That might not mean targeting R&D resources at reducing the accident rate for one minority group that experiences a higher-than-average malfunction rate. Here's a thought experiment to illustrate the principle I'm trying to get across: if we find out that members of a very small ethnic group suffer disproportionately from some deficiency in software that does a poor job at recognizing features common to them, and this leads to an extra 50 people dying each year, and that the resources it would take to fix this deficiency would allow the software to be improved to reduce the accident rate by 5%, leading to 5,000 fewer people dying each year, but only 5 fewer members of that ethnic group dying each year, should we target the resources at reducing the accident rate for the ethnic group, just because the group they find themselves in happens to be ethnic?
klyrs 7y ago
> So the algorithm wasn't tested on dark-skinned people. Is this really something new? No. Racism in the legal system, and our society overall, is not "new." However it's still a real problem in our society. As such, race- and gender-bias that's "accidentally" hard-coded into software sold to police is problematic. Every new product containing this flaw is news. > Why should it come as a shock that facial recognition product developers suffer from the same bias? At the very least, it should come as a shock that one of the biggest software development firms in the world is attacking critics in defense of buggy behavior. Especially since, as you note, this phenomenon is common knowledge and there's a simple and well-known workaround.
- Mirioron 7y ago
  Well, here's the question though: should she be naming and shaming the development company or instead the organization that uses the technology? I think the responsibility here lies solely on law enforcement as long as Amazon didn't lie. In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job.
  - klyrs 7y ago
    > ...should she be naming and shaming the development company... Yes, if you're offering a product for sale, it's fair for people to review that product in public. And that's not "naming and shaming," it's a critical review of an irresponsibly-developed product being sold to law-enforcement agencies. > In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job. Yes, that is also their responsibility. But where is the recourse? Law enforcement agencies have abysmal records of self-investigation, and the judicial system is unreliable at best, in holding law enforcement agencies accountable. The public has a right to know what technology is being used to police them. If you want to call investigative journalism "naming and shaming," then yes, absolutely, she made the ethical choice in speaking out.
- isoskeles 7y ago
  This is pretty low on the list of material problems that affect peoples' lives, in comparison to, for example, actual violence and crime. But I imagine it feels very satisfying to accuse society-at-large of "racism" and use scare quotes around "accidentally" to imply that some white men racistly hard-coded racism into a racist algorithm intended to ruin peoples' lives. I hope you got a good shot of endorphins in your brain by accusing people of intentionally building racist software.

Iolaum 7y ago

We should it be OK that everyone makes mistake A just because some people did it?

headalgorithm 7y ago

From the article: "Those disparities can sometimes be a matter of life or death: One recent study of the computer vision systems that enable self-driving cars to “see” the road shows they have a harder time detecting pedestrians with darker skin tones." Just think about the consequences of deploying such systems.

js2 7y ago
The story doesn’t cite the specific paper. But I think I submitted the paper last month: “Predictive Inequality in Object Detection” https://news.ycombinator.com/item?id=19379827 https://news.ycombinator.com/item?id=19379827 https://arxiv.org/abs/1902.11097 https://arxiv.org/abs/1902.11097 I’m not denying the conclusion of the paper, but it does have a lot of limitations. Also, Tesla notwithstanding, isn’t object detection primarily done via lidar?
jfk13 7y ago
I wonder if that's also true for human drivers. I certainly have a harder time detecting pedestrians wearing darker clothing, at least in some circumstances.
- skookumchuck 7y ago
  That's why road workers, cyclists, and joggers wear brightly colored reflective vests.
- munchbunny 7y ago
  There is also an important ethical difference between dark skin tones and dark clothing. I see the point you are making, but to the extent that an AI can do better than a human due to the physical aspect, for example, of dark skin reflecting less light at night, I think we should at least try.
tropo 7y ago
The computer vision systems could still beat real human vision systems. This could be by absolute numbers crashed into, or by having less bias than humans, or by both. It's not even a bias problem. It's a lighting, contrast, and camo problem. You could as well claim a bias between asphalt and concrete pavement, based on what skin tones are more visible with the pavement as a background.
- externalreality 7y ago
  > The computer vision systems could still beat real human vision systems. For pattern recognition, No, the computer vision system can't best a human.
  - tropo 7y ago
    I'm not saying it does today. I'm saying it could do so, while still having bias. The existence of bias doesn't make the computer worse than humans.
  - deogeo 7y ago
    But the entire self-driving system can. It is never tired, or distracted, has far fewer blind spots, and possibly LIDAR and IR to help it.
    - scottie_m 7y ago
      It can’t today, and until it proves to do so making claims about future tech in relation to present policy is a bad idea.
- int_19h 7y ago
  Suppose the system does beat real human vision by absolute numbers , but it turns out that it is objectively worse for someone who is not white, and the higher average is because the majority of people in that society are white. Is that acceptable?
  - deleted 7y ago
colllectorof 7y ago
So maybe we should not be using opaque ML algorithms for life-and-death situations to begin with? Instead of, you know, hoping that whomever is screwed by them has an active advocacy group in places like MIT.
- mikeash 7y ago
  Using NL for such things hasn’t been working too great, so it seems worthwhile to try for something better.
  - hprotagonist 7y ago
    such as?
    - mikeash 7y ago
      Are you asking for examples where NL hasn’t done well or for potential ways to do better?
      - hprotagonist 7y ago
        the latter. It’s relatively easy to say “stop doing X”; it’s harder to say “...and have you considered Y instead?”
        mikeash 7y ago
        I’d hope it was obvious from context that ML was the potential way to do better here.
        hprotagonist 7y ago
        I read you as saying that it was the thing doing poorly and that some superior object detection method was required. What that approach could be is totally unobvious to me.
sonnyblarney 7y ago
"Just think about the consequences of deploying such systems." Those systems are already widely deployed - they're called 'humans'. All vision systems are going to be more likely to confuse like colours. This problem is quite a bit different from the 'mugshot' problem, or a problem wherein there was a bias or lack of training data for certain samples. The author I think conflated a lot of the issues and just boiled it down to 'evil AI' which I don't think is the right thing to do. Issues of ethnic orientation are quite a bit different from systems that have trouble literally due to the colour of something.
maskymask 7y ago
I saw a talk by one of the authors of the pedestrian detection study that I think they are talking about, and their results did not seem support the scary racist self-driving car hypothesis that the study clearly implies (only marginal/statistically insignificant effects, with very little controls for confounding factors (I believe they controlled for bounding box size or something, but very little else)). With that said, fairness in ML/AI is a real problem, and some people are doing some really good/important work in this area. I am not familiar with Buolamwini's work, but I'm much more inclined to believe disparities in facial recognition than pedestrian detection where very little skin is visible and almost everything you're seeing is clothing.
deleted 7y ago
[deleted]

m0zg 7y ago

Edit: this factual description, written by a practitioner in the field of computer vision, of why the algorithms behave the way they do and what can be done about it, was downvoted and deliberately misinterpreted by the fake outrage mob. So I deleted it. I should have known better than to comment on a topic predisposed to attracting such people.

m0zg 7y ago
Edit: deleted
- danharaj 7y ago
  > you will get better at recognizing dark skinned people, at the expense of getting worse at recognizing light skinned ones, which generally constitute the majority in the real world Sure, buddy.
  - m0zg 7y ago
    In the US that's not up for debate. That's why you omitted "in most parts of the US" part of my message, I suppose, to manufacture outrage.
    - danharaj 7y ago
      You were making a value judgment about optimal utility. That's not a fact. They're just Internet Points, comrade. > I suppose, to manufacture outrage. You're the one who seems outraged =^)
      - m0zg 7y ago
        I made no judgment at all. I explained why the algorithms behave the way they do, and that there's currently no real way to make them behave better on darker skinned people without hurting accuracy overall, assuming the data distribution in the wild is not perfectly balanced, which it is not. That's literally all I said. Whatever you make up in your perfervid imagination is just that, your imagination, comrade.

mannykannot 7y ago

No-one is saying it is a shock - we're saying it is wrong and should be fixed. As for Amazon's self-serving response, it should be given the same degree of respect as any other statement by an entity that is not prepared to discuss it in an interview.

CryptoPunk 7y ago
Any product is going be a poorer fit for customers fitting a less common profile than those fitting a more common one. You will never get rid of that. Even if performance between races equalized due to a disproportionate amount of resources being expended on adapting the product to fit those belonging to minority racial groups, performance between unusual and common facial types or minority and majority ethnic groups within the races would exist. There will always be some minority group that the product fits less well with, because you can divide people into an infinite number of groups based on an infinite number of traits. This is a consequence of R&D resources being finite and the complexity of the world being unbounded. It's not sign of a moral failing or mis-placed priorities.
- externalreality 7y ago
  I agree with you. However, without trying to take your points down a slippery slope, your points don't apply to safety concerns. We can't have robo-cars hitting black people because the designers neglected to test the pattern recognition libraries on dark-skinned people. That, you can probably admit, is different from a bar of soap making peoples' skin a bit dry, or lotion not taking into account melanin, or the game designer neglecting the Linux community.
  - CryptoPunk 7y ago
    The accident rate should be decreased in the most efficient way possible, to save as many lives as possible. That might not mean targeting R&D resources at reducing the accident rate for one minority group that experiences a higher-than-average malfunction rate. Here's a thought experiment to illustrate the principle I'm trying to get across: if we find out that members of a very small ethnic group suffer disproportionately from some deficiency in software that does a poor job at recognizing features common to them, and this leads to an extra 50 people dying each year, and that the resources it would take to fix this deficiency would allow the software to be improved to reduce the accident rate by 5%, leading to 5,000 fewer people dying each year, but only 5 fewer members of that ethnic group dying each year, should we target the resources at reducing the accident rate for the ethnic group, just because the group they find themselves in happens to be ethnic?

klyrs 7y ago

> So the algorithm wasn't tested on dark-skinned people. Is this really something new? No. Racism in the legal system, and our society overall, is not "new." However it's still a real problem in our society. As such, race- and gender-bias that's "accidentally" hard-coded into software sold to police is problematic. Every new product containing this flaw is news. > Why should it come as a shock that facial recognition product developers suffer from the same bias? At the very least, it should come as a shock that one of the biggest software development firms in the world is attacking critics in defense of buggy behavior. Especially since, as you note, this phenomenon is common knowledge and there's a simple and well-known workaround.

Mirioron 7y ago
Well, here's the question though: should she be naming and shaming the development company or instead the organization that uses the technology? I think the responsibility here lies solely on law enforcement as long as Amazon didn't lie. In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job.
- klyrs 7y ago
  > ...should she be naming and shaming the development company... Yes, if you're offering a product for sale, it's fair for people to review that product in public. And that's not "naming and shaming," it's a critical review of an irresponsibly-developed product being sold to law-enforcement agencies. > In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job. Yes, that is also their responsibility. But where is the recourse? Law enforcement agencies have abysmal records of self-investigation, and the judicial system is unreliable at best, in holding law enforcement agencies accountable. The public has a right to know what technology is being used to police them. If you want to call investigative journalism "naming and shaming," then yes, absolutely, she made the ethical choice in speaking out.
isoskeles 7y ago
This is pretty low on the list of material problems that affect peoples' lives, in comparison to, for example, actual violence and crime. But I imagine it feels very satisfying to accuse society-at-large of "racism" and use scare quotes around "accidentally" to imply that some white men racistly hard-coded racism into a racist algorithm intended to ruin peoples' lives. I hope you got a good shot of endorphins in your brain by accusing people of intentionally building racist software.