5 ms·

This is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in

This is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in a way that the world isn't.

mlyle 6y ago
But it's completely on point, IMO: even if the model is biased against minorities in a way that the world actually is, it can be what we consider ethnically unfair. You're worse off lending in traditionally black neighborhoods; the question is, is it ethical to make a map of all of these traditionally black neighborhoods and refuse to lend there? A model that takes this into account would be biased in the same way the actual world is, but many would consider it deeply racist and unfair.
- karpierz 6y ago
  Sure, but "there are racial disparities in this domain, and therefore in the data" and "there aren't racial disparities in this domain, but there are in the data" are very different problems. The latter is the topic of discussion and is much less ambiguous than the former.
  - mlyle 6y ago
    There's also "there are no racial disparities in the training data, but the system still shows a response with racial disparity." ML has problems with all of these at times. The particular case here with the Obama picture is a particularly vivid illustration of one that can be used to bring attention to the overall problems.
- perl4ever 6y ago
  >in the same way the actual world is I feel like you've missed at least half of what I was trying to communicate, because you're still presenting arbitrary discrimination as reality-based. Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the true risks of individuals within the group. Because it's suboptimal, it can't be called an unique, objective reflection of the world. All correlations are inherently imperfect and therefore unfair; if they weren't you'd have an definite causal relationship. When you use one correlation, or set of correlations, maybe you have completely missed something that would be much better, not just from a moral perspective, but a business perspective. If you discriminate based on gender, and haven't considered age (hypothetically it being legal), maybe the latter would be a much better proxy for the real causal factors. Your gender model can be correct in itself, and yet from a completely amoral perspective be terrible and uncompetitive. The model is not "the way the world is" just because it's technically correct in isolation.
  - mlyle 6y ago
    > Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the people within the group who are better or worse risks. No model is perfect. Odds are you can better predict risk by (directly or indirectly) attaching bonuses or penalties to given races, all else being equal. e.g. Say, irrespective of all other measured quantities, people of race A are more likely to default because of other systemic racism against them. As a lender, it'd be completely rational to consider this and make "better" choices. And if you're not allowed to measure race, it'd be completely rational to find other variables that don't have an obvious causal relationship to credit risk, but predict race and thus have some information about credit risk. This is the exact kind of thing ML does. Then, in turn, this becomes self-reinforcing. Because other institutions discriminate against race A, the risks going forward of dealing with race A increase...
    - visarga 6y ago
      One possible approach is to make a model that besides its intended prediction also predicts race, and penalise its ability to predict race with probability larger than random. If the model has representations that can't predict race, then it will have to use non-biased features to make its predictions. But I guess in many situations this would degrade the accuracy and it will not be welcome from a business point of view.
      - mlyle 6y ago
        How would one do this? If you have a ML model with outputs classifying credit risk, and outputs classifying race, it's easy to learn coefficients that are bad at classifying race but still take race into account in classifying credit risk. That is, what's your loss function and how does it prevent race from being considered in the credit decision?
        ebalit 6y ago
        It's a setup that is somewhat similar to GANs (and even closer to a related method called Fader Networks): - a first network take the input data and return a representation A (like an embedding vector): let's called it the "censor network" - a second network take this embedding A as input and is trained to predict the class that should be censored (for example the gender of a person) : the "discriminator network" - a third network take the same embedding A as input and is trained to predict the real task of interest (for example the probability of credit default) : the "predictor network" The idea is that, by training the censor to make the discriminator fail (predict the wrong class) while making the predictor work, it will force the censor to learn a transformation of the input data that keeps the task related information in the embedding A, but removes the information correlated to the "censored class" (and that could be used to discriminate). Here's a reference about this kind of methods, but it's still an active domain of study in ML and there are many papers that followed this one: https://arxiv.org/pdf/1801.07593.pdf https://arxiv.org/pdf/1801.07593.pdf
- fny 6y ago
  It is not legal to make such a map. In practice, production actuarial systems for loans need to be interpretable and auditable, so this argument is moot.
  - danielheath 6y ago
    In practice, multiple advertisers have been in trouble for breaking this class of rule in the past 2 years. That's not some long-forgotten problem, it's current. If neural nets provide a way maintain plausible deniability while breaking the rules, it's going to get used.
    - andreig2191 6y ago
      I'm not sure neural nets provide plausible deniability. The NN can be proven to be discriminatory.
deleted 6y ago
[deleted]

This is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in a way that the world isn't.

mlyle 6y ago

But it's completely on point, IMO: even if the model is biased against minorities in a way that the world actually is, it can be what we consider ethnically unfair. You're worse off lending in traditionally black neighborhoods; the question is, is it ethical to make a map of all of these traditionally black neighborhoods and refuse to lend there? A model that takes this into account would be biased in the same way the actual world is, but many would consider it deeply racist and unfair.

karpierz 6y ago
Sure, but "there are racial disparities in this domain, and therefore in the data" and "there aren't racial disparities in this domain, but there are in the data" are very different problems. The latter is the topic of discussion and is much less ambiguous than the former.
- mlyle 6y ago
  There's also "there are no racial disparities in the training data, but the system still shows a response with racial disparity." ML has problems with all of these at times. The particular case here with the Obama picture is a particularly vivid illustration of one that can be used to bring attention to the overall problems.
perl4ever 6y ago
>in the same way the actual world is I feel like you've missed at least half of what I was trying to communicate, because you're still presenting arbitrary discrimination as reality-based. Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the true risks of individuals within the group. Because it's suboptimal, it can't be called an unique, objective reflection of the world. All correlations are inherently imperfect and therefore unfair; if they weren't you'd have an definite causal relationship. When you use one correlation, or set of correlations, maybe you have completely missed something that would be much better, not just from a moral perspective, but a business perspective. If you discriminate based on gender, and haven't considered age (hypothetically it being legal), maybe the latter would be a much better proxy for the real causal factors. Your gender model can be correct in itself, and yet from a completely amoral perspective be terrible and uncompetitive. The model is not "the way the world is" just because it's technically correct in isolation.
- mlyle 6y ago
  > Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the people within the group who are better or worse risks. No model is perfect. Odds are you can better predict risk by (directly or indirectly) attaching bonuses or penalties to given races, all else being equal. e.g. Say, irrespective of all other measured quantities, people of race A are more likely to default because of other systemic racism against them. As a lender, it'd be completely rational to consider this and make "better" choices. And if you're not allowed to measure race, it'd be completely rational to find other variables that don't have an obvious causal relationship to credit risk, but predict race and thus have some information about credit risk. This is the exact kind of thing ML does. Then, in turn, this becomes self-reinforcing. Because other institutions discriminate against race A, the risks going forward of dealing with race A increase...
  - visarga 6y ago
    One possible approach is to make a model that besides its intended prediction also predicts race, and penalise its ability to predict race with probability larger than random. If the model has representations that can't predict race, then it will have to use non-biased features to make its predictions. But I guess in many situations this would degrade the accuracy and it will not be welcome from a business point of view.
    - mlyle 6y ago
      How would one do this? If you have a ML model with outputs classifying credit risk, and outputs classifying race, it's easy to learn coefficients that are bad at classifying race but still take race into account in classifying credit risk. That is, what's your loss function and how does it prevent race from being considered in the credit decision?
      - ebalit 6y ago
        It's a setup that is somewhat similar to GANs (and even closer to a related method called Fader Networks): - a first network take the input data and return a representation A (like an embedding vector): let's called it the "censor network" - a second network take this embedding A as input and is trained to predict the class that should be censored (for example the gender of a person) : the "discriminator network" - a third network take the same embedding A as input and is trained to predict the real task of interest (for example the probability of credit default) : the "predictor network" The idea is that, by training the censor to make the discriminator fail (predict the wrong class) while making the predictor work, it will force the censor to learn a transformation of the input data that keeps the task related information in the embedding A, but removes the information correlated to the "censored class" (and that could be used to discriminate). Here's a reference about this kind of methods, but it's still an active domain of study in ML and there are many papers that followed this one: https://arxiv.org/pdf/1801.07593.pdf https://arxiv.org/pdf/1801.07593.pdf
fny 6y ago
It is not legal to make such a map. In practice, production actuarial systems for loans need to be interpretable and auditable, so this argument is moot.
- danielheath 6y ago
  In practice, multiple advertisers have been in trouble for breaking this class of rule in the past 2 years. That's not some long-forgotten problem, it's current. If neural nets provide a way maintain plausible deniability while breaking the rules, it's going to get used.
  - andreig2191 6y ago
    I'm not sure neural nets provide plausible deniability. The NN can be proven to be discriminatory.

deleted 6y ago

[deleted]