4 ms·

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in les

by gfodor 8y ago

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class? If we're being honest, a system only needs to be in a decision-making capacity for discriminatory behavior to be scrutinized, since in many cases human operators will not be able to identify the specific features being used to make decisions about people -- the features could be highly correlated with some subpopulation of protected class. If you take that to be true, the question reduces onto what decision-making roles ML algorithms have that could be discriminatory, and it's hard to argue this is not a massive part of their current and expected roles. I think this is going to be a long, winding ethical nightmare that is probably just getting started by human-digestible examples such as these. One can imagine things like this one being looked back on as quaint in the naivety to which we assume we can understand these systems. Where do we draw the line, and how much control do we give up to an optimization function? Surely there is a balance -- how do we categorize and made good decisions around this? As far as I know, a cohesive ethical framework around this is pretty much non-existent -- the current regime is simply "someone speaks up when something absurdly and overtly bad happens."

abdullahkhalids 8y ago
> What about cases where the features just merely correlated with a subpopulation of a protected class? This is just Simpson's paradox [1] which is notoriously hard to identify because you have to compare the overall with the breakdown. As you say, current-AI probably already has such biases. [1] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
jakelazaroff 8y ago
> What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.
- darawk 8y ago
  Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women? If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?
  - gfodor 8y ago
    Separate it one step further -- hiring decisions are all too easy to pin as discriminatory. What if a site shows ads for a special offer on protein shakes for people with higher bicep mass since they are in the target market? Is that discriminating against women?
    - rrcaptain 8y ago
      How the hell are you targeting ads with "bicep mass"?
  - jakelazaroff 8y ago
    Underlying your examples is the implication that a preference shouldn't be considered discriminatory if the trait being selected for correlates with fitness. I agree with this position! What I don't agree with is the assumption that, in this case, the preferred traits do correlate with fitness, since there's at least one — gender — for which this model is biased even though it has no apparent correlation.
    - darawk 8y ago
      Ya, I just mean to say that uncorrelation with fitness is an important qualifier.
darawk 8y ago
All of that is true, but I think the most important question is: compared to what? ML is substantially more transparent than human decision makers. Human decision makers will actively lie to you. ML is a major step forward in correcting these sorts of biases, by making interpretable (relative to humans) models in the first place.
Bartweiss 8y ago
> the features are just merely correlated with a subpopulation of a protected class The article notes that Amazon's system rated down grads from two all-women's schools. But it immediately occurs to me to wonder what the algorithm did with candidates from heavily gender-imbalanced schools, which could be much harder to spot. RPI's Computer Science department is about 85% male, while CMU's is just over 50% male. CMU's CS department is also considered one of the best in the world, and presumably any functional algorithm that cared about alma mater would respond to that. So if the bias ends up being "because of CMU's gender ratio, CMU grads with gender-unclear resumes are advantaged slightly less than otherwise would be", how on earth would someone spot that? Once you're looking for it, you could potentially retrain with some data set like "RPI resumes, but we adjusted their gendered-words rate" and see if you get a different outcome on your test set. But that's both a labor intensive task, and one that's only approachable once you already know what you're looking for. And even if you do see a change, you'd still have to tease it out from a dozen other hypotheses like "certain schools have more organizations with gendered names, and the algorithm can't tell that those organizations are a proxy for school". Of course, the counterpoint is that human decisions can't be scrutinized any better, and it's not entirely clear they're less arbitrary or more ethical. At a certain point algorithmic approaches are being scrutinized because they're slightly transparent and testable, so running them on a range of counterfactuals or breaking down their choices is hard rather than impossible. I suspect that's true, but it doesn't really comfort me - humans at least tend to misbehave along certain predictable axes we can try to mitigate, while ML systems can blindside us with all sorts of new and unexpected forms of badness.
- grogenaut 8y ago
  A subtle point you may have missed, amazon knew about and accounted for the gender bias, the scrapped it because of all of the biases that they couldn't identify and were leery of. Most of your suggestions seem to be solving for the known biases, which I believe they did. Also knowing some people who worked on this, they were VERY cognizant of re-encoding biases from the start of the project, it was one of the main reasons they thought the project might fail.
  - 8y ago
jhfhhhf 8y ago
Protected classes? Can I be a protected class? I don't think having protected classes is a good thing, tbh...

by gfodor 8y ago

abdullahkhalids 8y ago

> What about cases where the features just merely correlated with a subpopulation of a protected class? This is just Simpson's paradox [1] which is notoriously hard to identify because you have to compare the overall with the breakdown. As you say, current-AI probably already has such biases. [1] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox

jakelazaroff 8y ago

> What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.

darawk 8y ago
Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women? If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?
- gfodor 8y ago
  Separate it one step further -- hiring decisions are all too easy to pin as discriminatory. What if a site shows ads for a special offer on protein shakes for people with higher bicep mass since they are in the target market? Is that discriminating against women?
  - rrcaptain 8y ago
    How the hell are you targeting ads with "bicep mass"?
- jakelazaroff 8y ago
  Underlying your examples is the implication that a preference shouldn't be considered discriminatory if the trait being selected for correlates with fitness. I agree with this position! What I don't agree with is the assumption that, in this case, the preferred traits do correlate with fitness, since there's at least one — gender — for which this model is biased even though it has no apparent correlation.
  - darawk 8y ago
    Ya, I just mean to say that uncorrelation with fitness is an important qualifier.

darawk 8y ago

All of that is true, but I think the most important question is: compared to what? ML is substantially more transparent than human decision makers. Human decision makers will actively lie to you. ML is a major step forward in correcting these sorts of biases, by making interpretable (relative to humans) models in the first place.

Bartweiss 8y ago

> the features are just merely correlated with a subpopulation of a protected class The article notes that Amazon's system rated down grads from two all-women's schools. But it immediately occurs to me to wonder what the algorithm did with candidates from heavily gender-imbalanced schools, which could be much harder to spot. RPI's Computer Science department is about 85% male, while CMU's is just over 50% male. CMU's CS department is also considered one of the best in the world, and presumably any functional algorithm that cared about alma mater would respond to that. So if the bias ends up being "because of CMU's gender ratio, CMU grads with gender-unclear resumes are advantaged slightly less than otherwise would be", how on earth would someone spot that? Once you're looking for it, you could potentially retrain with some data set like "RPI resumes, but we adjusted their gendered-words rate" and see if you get a different outcome on your test set. But that's both a labor intensive task, and one that's only approachable once you already know what you're looking for. And even if you do see a change, you'd still have to tease it out from a dozen other hypotheses like "certain schools have more organizations with gendered names, and the algorithm can't tell that those organizations are a proxy for school". Of course, the counterpoint is that human decisions can't be scrutinized any better, and it's not entirely clear they're less arbitrary or more ethical. At a certain point algorithmic approaches are being scrutinized because they're slightly transparent and testable, so running them on a range of counterfactuals or breaking down their choices is hard rather than impossible. I suspect that's true, but it doesn't really comfort me - humans at least tend to misbehave along certain predictable axes we can try to mitigate, while ML systems can blindside us with all sorts of new and unexpected forms of badness.

grogenaut 8y ago
A subtle point you may have missed, amazon knew about and accounted for the gender bias, the scrapped it because of all of the biases that they couldn't identify and were leery of. Most of your suggestions seem to be solving for the known biases, which I believe they did. Also knowing some people who worked on this, they were VERY cognizant of re-encoding biases from the start of the project, it was one of the main reasons they thought the project might fail.
- 8y ago

jhfhhhf 8y ago

Protected classes? Can I be a protected class? I don't think having protected classes is a good thing, tbh...