5 ms·

> diversity & inclusion review Is this tongue-in-cheek, or are you serious? Poe's law and all that.

kevingadd 5y ago
If you're publishing a dataset in the terabytes it does actually make sense to at least do a pass over it and make sure the data you're using isn't skewed in any undesirable way that would cause problems down the road. For example, if you're releasing 5tb of face photos for training facial recognition nets, it would certainly be a problem if all the faces are white women or asian men - the result would probably be over-fit and not perform as well for people in other categories. It would be correct to call that a diversity/inclusion issue. Privacy and accessibility reviews serve similar purposes there, you're reducing risk by checking for these various problems and ideally they also spot ways to improve the quality of your outcomes.
- dekhn 5y ago
  the 5tb was performance data collected from servers
  - kevingadd 5y ago
    Sounds like the reviewer would glance at it for 5 seconds and say 'ok'
    - drexlspivey 5y ago
      What if some servers were excluded?
- murph-almighty 5y ago
  It's common in fintech for data/ML models to go through similar overview. If you happen to disenfranchise a set of people because your model said not to lend to them, you risk legal jeopardy. To clarify, I think it's good that this is a practice.
  - londons_explore 5y ago
    A review doesn't necessarily mean you need to resolve all diversity/inclusion issues. It can merely require that you identify the issues and understand the risks of not resolving them.
  - charcircuit 5y ago
    The whole point of the model is to find who not to lend to. You are always going to exclude people by definition.
    - murph-almighty 5y ago
      I should clarify, the point is to not discriminate against a protected class.
    - criley2 5y ago
      Tell that to the legislators and prosecutors who create laws and enforce laws against you.
    - myownpetard 5y ago
      There are so many ways you can accidentally systematize racism in software like automated lending. In the past there were explicitly racist policies like redlining. This leads to a historical data set of loan denials to people in specific racial group. If that group has other traits that correlate to their race, e.g. the neighborhood they live in then you could presumably have a model that doesn't explicitly have race as a feature but uses that historical data and some subset of racially correlated features and as a result disproportionately excludes people of that race.
      - belorn 5y ago
        I am not sure how one would remove all ageism, sexism, racism, classism, title-ism, and so on from lending. The whole concept is about making a prediction about the future with sub optimal information, guessing who will default on a loan and who won't. Same goes with insurance. I have been pretty tempted to lie about where I live in order to reduce my insurance costs. It would reduce the insurance cost by half. It seems pretty disproportionately harsh that I should get lumped together with the people who simply happen to live around me. Is it possible to make predictions illegal if they are based on historical data from other than the individual customer?
smartician 5y ago
Partly tongue-in-cheek. These review processes exist, but whether they're required or not depends on the product area and type of project.
rodgerd 5y ago
Perhaps Google don't want to be in the news for identifying dark-skinned people as monkeys again?
- jjeaff 5y ago
  I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company. They could have probably benefited from a diversity and inclusion review. Heck, employing a single brown engineer or even QA engineer probably would have been enough to notice that before launch.
  - alasdair_ 5y ago
    Microsoft Kinect also had this issue. I know the one black engineer (that had little to do with the project) that repeatedly got pulled in to see if the test system worked with non-white people. He left soon after.
    - shadowgovt 5y ago
      I wonder if the company failed to give him credit, responsibility, or compensation commensurate with his value to the project?
  - saltminer 5y ago
    > I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company It was HP https://www.youtube.com/watch?v=t4DT3tQqgRM https://www.youtube.com/watch?v=t4DT3tQqgRM
  - NaturalPhallacy 5y ago
    People jump to the racism conclusion here, but it's really just a contrast issue. Detecting eyes, for example is simply easier with lighter skin. Light skinned black people read just fine, and super tanned white people are harder to read. It's literally contrast (light) detection, not racism. But because the media keeps everybody primed for racism to stave off the necessary class power rebalancing, everyone jumps to racism.

If you're publishing a dataset in the terabytes it does actually make sense to at least do a pass over it and make sure the data you're using isn't skewed in any undesirable way that would cause problems down the road. For example, if you're releasing 5tb of face photos for training facial recognition nets, it would certainly be a problem if all the faces are white women or asian men - the result would probably be over-fit and not perform as well for people in other categories. It would be correct to call that a diversity/inclusion issue. Privacy and accessibility reviews serve similar purposes there, you're reducing risk by checking for these various problems and ideally they also spot ways to improve the quality of your outcomes.

dekhn 5y ago
the 5tb was performance data collected from servers
- kevingadd 5y ago
  Sounds like the reviewer would glance at it for 5 seconds and say 'ok'
  - drexlspivey 5y ago
    What if some servers were excluded?
murph-almighty 5y ago
It's common in fintech for data/ML models to go through similar overview. If you happen to disenfranchise a set of people because your model said not to lend to them, you risk legal jeopardy. To clarify, I think it's good that this is a practice.
- londons_explore 5y ago
  A review doesn't necessarily mean you need to resolve all diversity/inclusion issues. It can merely require that you identify the issues and understand the risks of not resolving them.
- charcircuit 5y ago
  The whole point of the model is to find who not to lend to. You are always going to exclude people by definition.
  - murph-almighty 5y ago
    I should clarify, the point is to not discriminate against a protected class.
  - criley2 5y ago
    Tell that to the legislators and prosecutors who create laws and enforce laws against you.
  - myownpetard 5y ago
    There are so many ways you can accidentally systematize racism in software like automated lending. In the past there were explicitly racist policies like redlining. This leads to a historical data set of loan denials to people in specific racial group. If that group has other traits that correlate to their race, e.g. the neighborhood they live in then you could presumably have a model that doesn't explicitly have race as a feature but uses that historical data and some subset of racially correlated features and as a result disproportionately excludes people of that race.
    - belorn 5y ago
      I am not sure how one would remove all ageism, sexism, racism, classism, title-ism, and so on from lending. The whole concept is about making a prediction about the future with sub optimal information, guessing who will default on a loan and who won't. Same goes with insurance. I have been pretty tempted to lie about where I live in order to reduce my insurance costs. It would reduce the insurance cost by half. It seems pretty disproportionately harsh that I should get lumped together with the people who simply happen to live around me. Is it possible to make predictions illegal if they are based on historical data from other than the individual customer?

smartician 5y ago

Partly tongue-in-cheek. These review processes exist, but whether they're required or not depends on the product area and type of project.

rodgerd 5y ago

Perhaps Google don't want to be in the news for identifying dark-skinned people as monkeys again?

jjeaff 5y ago
I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company. They could have probably benefited from a diversity and inclusion review. Heck, employing a single brown engineer or even QA engineer probably would have been enough to notice that before launch.
- alasdair_ 5y ago
  Microsoft Kinect also had this issue. I know the one black engineer (that had little to do with the project) that repeatedly got pulled in to see if the test system worked with non-white people. He left soon after.
  - shadowgovt 5y ago
    I wonder if the company failed to give him credit, responsibility, or compensation commensurate with his value to the project?
- saltminer 5y ago
  > I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company It was HP https://www.youtube.com/watch?v=t4DT3tQqgRM https://www.youtube.com/watch?v=t4DT3tQqgRM
- NaturalPhallacy 5y ago
  People jump to the racism conclusion here, but it's really just a contrast issue. Detecting eyes, for example is simply easier with lighter skin. Light skinned black people read just fine, and super tanned white people are harder to read. It's literally contrast (light) detection, not racism. But because the media keeps everybody primed for racism to stave off the necessary class power rebalancing, everyone jumps to racism.