5 ms·

This will further marginalize AI ethics and can only be seen as a good thing. NPR did yet another piece on the topic last weekend and interviewed an AI ethics

This will further marginalize AI ethics and can only be seen as a good thing. NPR did yet another piece on the topic last weekend and interviewed an AI ethics expert. When, once again, I’m being told that the reason a system has a hard time identifying the features of black faces is systemic racism, and doesn’t even mention the optic and sensor issues, I think the field needs a reboot.

nerdponx 6y ago
It is precisely systemic racism. People who do not have dark skin make a system. It turns out that the system performs poorly on people with dark skin, and possibly causes discriminatory outcomes. Perhaps the training data didn't have enough dark-skinned people. Or perhaps the optics and sensors didn't work well. The point is that the outcome is biased against a certain group of people, and it's (edit: partly) because that affected group of people lacks representation in the group of people developing the system (edit: , because if they were represented they would have flagged the issue and it would have all been presented and handled differently). No, it does not imply that the individual researchers are racist as such.
- sprayk 6y ago
  Is there a way to prove that it is not caused by systemic racism? Having some unfalsifiable quality is pretty useless, and if it is the sort of thing that's never meant to end then that feels more like religion.
  - nerdponx 6y ago
    It would depend on the details of the particular situation (was there a specific news story associated with the GP?), but you'd have to show that people with the skin color in question signed off on the product. It's one thing if you produce a new model and present it at a conference, and one of the conclusion of your paper is that "this model does great on light-skinned faces but performs poorly on dark-skinned faces, we need to make improvements in optics and low-light sensing to bring up the accuracy on dark-skinned people." That's not racist, that's doing science. It's another if you are a for-profit image recognition company, or if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc. We are currently in a world where the latter happens quite often. It's not just a race thing; there all kinds of inadequate "AI" systems being pushed on people that don't really work as advertised, to the detriment of some group or another (sometimes to everyone). But when it becomes a race thing, it's especially distressing to a lot of people, because Europe and the USA have long and grim histories of racism that extend up to and including the present day.
    - sprayk 6y ago
      > ... people with the skin color in question signed off on the product. > ... if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc Neither of these would rule out systemic racism in either case, though, since, as you said, the problem is that "the outcome is biased against a certain group of people". Regardless of whether there were BIPOC people signing off or whether you had a perfect training set/etc, the only thing that matters is the outcome.
      - nerdponx 6y ago
        That's right. My example stated two very different outcomes. One outcome: an academic result is published. Other outcomes: something is put into production.
        sprayk 6y ago
        so you agree that attributing an outcome to systemic racism is unfalsifiable?
        nerdponx 6y ago
        I see what you're saying. Mind you, I'm not a sociologist, so anything I say here is going to be a suboptimal paraphrasing. Systemic racism is not a cause of something, but a description of a state of existence. Systemic racism is itself the existence of inequitable outcomes in the context of a society that was literally founded on overt racism. That something "is" systemic racism is falsifiable by demonstrating that one or both of those criteria is not met. So this particular outcome will always be definitionally systemic racism until the outcome is no longer inequitable, or the USA has moved meaningfully past its white supremacist origins to the point where individual cases of accidental bias are truly individual cases and not part of a pattern. Personally, I am okay with that. Ceteris non paribus sunt.
  - themolecularman 6y ago
    Agreed, but I've found that when discussing these things logic isn't often taken into account anymore due to matters of sensitivity and bias. It's frustrating.
- SpicyLemonZest 6y ago
  I understand what you're saying, but it sounds like you're aware of how prone this term is to misinterpretation. Virtually everyone outside of a particular small subculture understands "systemic racism" to be an accusation of racism. It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack?
  - DenisM 6y ago
    > It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack? Perhaps they aren't.
    - nerdponx 6y ago
      Anecdata: At work one time I created a machine learning "algorithm" that turned out to be systemically biased in a way that was both unintended and distasteful. I'm sure I've done it other times, but this is the one time that we actually realized it. The issue was flagged internally and the model was improved until we felt like we were no longer producing bigoted results. I certainly did not take offense at the suggestion that my model might be racist. On the contrary, I considered it a very important and serious problem that needed to be resolved. It's worth mentioning that part of the reason we even caught the problem is that some of my colleagues at the time would have been affected by the adverse outcome, had they been subject to the judgement of the model. If I didn't personally work with people who fell into the affected group, I might never have noticed the problem in the first place.
  - deleted 6y ago
    [deleted]
- Invictus0 6y ago
  > The point is that the outcome is biased against a certain group of people, and it's because that affected group of people lacks representation in the group of people developing the system. Arguably the real reason is that dark skinned people look more like gorillas than light skinned people, and distinguishing the two is a non-trivial technical problem. Would you still call it racist if an entirely dark skinned team of researchers built the system? If it were anything else, we would just call it "buggy new technology" and move on; but because this touches a sensitive race issue, it imports all the nonsense on that issue and then hampers innovation in the field.
  - jdgoesmarching 6y ago
    “It’s not racism, dark skinned people just look like gorillas.” When your “bugs” are consistently found to be discriminatory, then maybe there’s an underlying issue. Bias affects all of us, devs need to stop being so defensive about it.
    - Udik 6y ago
      > When your “bugs” are consistently found to be discriminatory Unreliable automatic face recognition is a bug or a feature? I'm not sure. If the parts were inverted, I' pretty sure someone would say "Hey, whites are privileged as the face recognition systems are unreliable on them, so they cannot be tracked by the surveillance state- this is systemic racism!"
      - deleted 6y ago
        [deleted]
- deleted 6y ago
  [deleted]
- HideousKojima 6y ago
  Alternative take: Computers can most easily pick out facial features when there is significant color contrast, shadows cast on white skin create greater contrast than it can with darker skin, and the systems end up inherently working better on people with lighter skin due to basic math and physics. Why hasn't some company out there designed a powerful system that works well with darker skinned people? It would be trumpeted by the tech press all over if they did, instead we just see articles about companies like Google manually tweaking the outcomes to appease the people complaining about racism.
  - nerdponx 6y ago
    A system that works poorly on dark-skinned people would be a nonstarter in most parts of Africa and South Asia. But in North America it's hailed as highly successful and put into production, despite some 44 million US citizens being Black or African, with demographic concentrations as high as 70-80% in some cities.
    - HideousKojima 6y ago
      From looking at articles that actually give hard numbers, misidentification for minorities is something like 5 to 10 times more common, with a 1 in 1000 error rate for black women for example: https://www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/ https://www.wired.com/story/best-algorithms-struggle-recogni... Being able to identify 999 out of 1,000 black women accurately is highly successful. Just because it gets 9,999 out of 10,000 on white women doesn't make 999 out of 1,000 a failure. Mind you I'm opposed to the tech in general because it will inevitably be used to create dystopian hellholes. But not because I falsely correlate different error margins for different races with "structural racism."
uniqueid 6y ago
This will further marginalize AI ethics and can only be seen as a good thing. I don't think marginalizing AI ethics 'can only' be seen as a good thing. Surely a few people don't want to be turned into paperclips.
- SpicyLemonZest 6y ago
  In recent years, "AI ethics" generally refers to the problem of building regression models whose mechanics and outputs follow a certain set of political sensibilities. People investigating the problem you're describing typically identify their work as "AI safety".
sudosysgen 6y ago
The reason why we don't care and still use passive optics to do face recognition when people with black skin exist is literally systematic racism.
- HideousKojima 6y ago
  Or maybe it's just one of the simpler and least computationally expensive ways to do it? No systemic racism needed (unless you're stretching the meaning of the word beyond all recognition)
  - sudosysgen 6y ago
    I'm not stretching the word beyond anything past it's intended meaning. If passive optical image based image recognition was as performant for most people as it is in general for black people it wouldn't be acceptable. The fact that this trade-off is accepted is systematic racism - the systematic incentives of implementing facial recognition leads to discrimination.
    - hntrader 6y ago
      Do you know what the classification error is for different ethnicities? Or are you just basing this on a small handful of misclassifications that you've seen in the news?
      - sudosysgen 6y ago
        We're talking specifically about passive vision based facial recognition, right? I've personally seen such systems that simply do not have the dynamic range to be able to, in the same exposure, extract a sufficient amount of detail from black skinned faces and white skinned faces. I'm sure there are systems that can, at a much higher cost. But largely there are many that simply fail.

nerdponx 6y ago

It is precisely systemic racism. People who do not have dark skin make a system. It turns out that the system performs poorly on people with dark skin, and possibly causes discriminatory outcomes. Perhaps the training data didn't have enough dark-skinned people. Or perhaps the optics and sensors didn't work well. The point is that the outcome is biased against a certain group of people, and it's (edit: partly) because that affected group of people lacks representation in the group of people developing the system (edit: , because if they were represented they would have flagged the issue and it would have all been presented and handled differently). No, it does not imply that the individual researchers are racist as such.

sprayk 6y ago
Is there a way to prove that it is not caused by systemic racism? Having some unfalsifiable quality is pretty useless, and if it is the sort of thing that's never meant to end then that feels more like religion.
- nerdponx 6y ago
  It would depend on the details of the particular situation (was there a specific news story associated with the GP?), but you'd have to show that people with the skin color in question signed off on the product. It's one thing if you produce a new model and present it at a conference, and one of the conclusion of your paper is that "this model does great on light-skinned faces but performs poorly on dark-skinned faces, we need to make improvements in optics and low-light sensing to bring up the accuracy on dark-skinned people." That's not racist, that's doing science. It's another if you are a for-profit image recognition company, or if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc. We are currently in a world where the latter happens quite often. It's not just a race thing; there all kinds of inadequate "AI" systems being pushed on people that don't really work as advertised, to the detriment of some group or another (sometimes to everyone). But when it becomes a race thing, it's especially distressing to a lot of people, because Europe and the USA have long and grim histories of racism that extend up to and including the present day.
  - sprayk 6y ago
    > ... people with the skin color in question signed off on the product. > ... if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc Neither of these would rule out systemic racism in either case, though, since, as you said, the problem is that "the outcome is biased against a certain group of people". Regardless of whether there were BIPOC people signing off or whether you had a perfect training set/etc, the only thing that matters is the outcome.
    - nerdponx 6y ago
      That's right. My example stated two very different outcomes. One outcome: an academic result is published. Other outcomes: something is put into production.
      - sprayk 6y ago
        so you agree that attributing an outcome to systemic racism is unfalsifiable?
        nerdponx 6y ago
        I see what you're saying. Mind you, I'm not a sociologist, so anything I say here is going to be a suboptimal paraphrasing. Systemic racism is not a cause of something, but a description of a state of existence. Systemic racism is itself the existence of inequitable outcomes in the context of a society that was literally founded on overt racism. That something "is" systemic racism is falsifiable by demonstrating that one or both of those criteria is not met. So this particular outcome will always be definitionally systemic racism until the outcome is no longer inequitable, or the USA has moved meaningfully past its white supremacist origins to the point where individual cases of accidental bias are truly individual cases and not part of a pattern. Personally, I am okay with that. Ceteris non paribus sunt.
- themolecularman 6y ago
  Agreed, but I've found that when discussing these things logic isn't often taken into account anymore due to matters of sensitivity and bias. It's frustrating.
SpicyLemonZest 6y ago
I understand what you're saying, but it sounds like you're aware of how prone this term is to misinterpretation. Virtually everyone outside of a particular small subculture understands "systemic racism" to be an accusation of racism. It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack?
- DenisM 6y ago
  > It would be like calling a correctness bug "systemic fraud" - how are the authors of the system supposed to avoid taking that as a personal attack? Perhaps they aren't.
  - nerdponx 6y ago
    Anecdata: At work one time I created a machine learning "algorithm" that turned out to be systemically biased in a way that was both unintended and distasteful. I'm sure I've done it other times, but this is the one time that we actually realized it. The issue was flagged internally and the model was improved until we felt like we were no longer producing bigoted results. I certainly did not take offense at the suggestion that my model might be racist. On the contrary, I considered it a very important and serious problem that needed to be resolved. It's worth mentioning that part of the reason we even caught the problem is that some of my colleagues at the time would have been affected by the adverse outcome, had they been subject to the judgement of the model. If I didn't personally work with people who fell into the affected group, I might never have noticed the problem in the first place.
- deleted 6y ago
  [deleted]
Invictus0 6y ago
> The point is that the outcome is biased against a certain group of people, and it's because that affected group of people lacks representation in the group of people developing the system. Arguably the real reason is that dark skinned people look more like gorillas than light skinned people, and distinguishing the two is a non-trivial technical problem. Would you still call it racist if an entirely dark skinned team of researchers built the system? If it were anything else, we would just call it "buggy new technology" and move on; but because this touches a sensitive race issue, it imports all the nonsense on that issue and then hampers innovation in the field.
- jdgoesmarching 6y ago
  “It’s not racism, dark skinned people just look like gorillas.” When your “bugs” are consistently found to be discriminatory, then maybe there’s an underlying issue. Bias affects all of us, devs need to stop being so defensive about it.
  - Udik 6y ago
    > When your “bugs” are consistently found to be discriminatory Unreliable automatic face recognition is a bug or a feature? I'm not sure. If the parts were inverted, I' pretty sure someone would say "Hey, whites are privileged as the face recognition systems are unreliable on them, so they cannot be tracked by the surveillance state- this is systemic racism!"
    - deleted 6y ago
      [deleted]
deleted 6y ago
[deleted]
HideousKojima 6y ago
Alternative take: Computers can most easily pick out facial features when there is significant color contrast, shadows cast on white skin create greater contrast than it can with darker skin, and the systems end up inherently working better on people with lighter skin due to basic math and physics. Why hasn't some company out there designed a powerful system that works well with darker skinned people? It would be trumpeted by the tech press all over if they did, instead we just see articles about companies like Google manually tweaking the outcomes to appease the people complaining about racism.
- nerdponx 6y ago
  A system that works poorly on dark-skinned people would be a nonstarter in most parts of Africa and South Asia. But in North America it's hailed as highly successful and put into production, despite some 44 million US citizens being Black or African, with demographic concentrations as high as 70-80% in some cities.
  - HideousKojima 6y ago
    From looking at articles that actually give hard numbers, misidentification for minorities is something like 5 to 10 times more common, with a 1 in 1000 error rate for black women for example: https://www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/ https://www.wired.com/story/best-algorithms-struggle-recogni... Being able to identify 999 out of 1,000 black women accurately is highly successful. Just because it gets 9,999 out of 10,000 on white women doesn't make 999 out of 1,000 a failure. Mind you I'm opposed to the tech in general because it will inevitably be used to create dystopian hellholes. But not because I falsely correlate different error margins for different races with "structural racism."

uniqueid 6y ago

This will further marginalize AI ethics and can only be seen as a good thing. I don't think marginalizing AI ethics 'can only' be seen as a good thing. Surely a few people don't want to be turned into paperclips.

SpicyLemonZest 6y ago
In recent years, "AI ethics" generally refers to the problem of building regression models whose mechanics and outputs follow a certain set of political sensibilities. People investigating the problem you're describing typically identify their work as "AI safety".

sudosysgen 6y ago

The reason why we don't care and still use passive optics to do face recognition when people with black skin exist is literally systematic racism.

HideousKojima 6y ago
Or maybe it's just one of the simpler and least computationally expensive ways to do it? No systemic racism needed (unless you're stretching the meaning of the word beyond all recognition)
- sudosysgen 6y ago
  I'm not stretching the word beyond anything past it's intended meaning. If passive optical image based image recognition was as performant for most people as it is in general for black people it wouldn't be acceptable. The fact that this trade-off is accepted is systematic racism - the systematic incentives of implementing facial recognition leads to discrimination.
  - hntrader 6y ago
    Do you know what the classification error is for different ethnicities? Or are you just basing this on a small handful of misclassifications that you've seen in the news?
    - sudosysgen 6y ago
      We're talking specifically about passive vision based facial recognition, right? I've personally seen such systems that simply do not have the dynamic range to be able to, in the same exposure, extract a sufficient amount of detail from black skinned faces and white skinned faces. I'm sure there are systems that can, at a much higher cost. But largely there are many that simply fail.