5 ms·

Google has actually released a paper on codifying what discrimination looks like for lending (https://arxiv.org/abs/1610.02413 https://arxiv.org/abs/1610.02413

by dlss 10y ago

Google has actually released a paper on codifying what discrimination looks like for lending (https://arxiv.org/abs/1610.02413 https://arxiv.org/abs/1610.02413 + blog post: https://research.googleblog.com/2016/10/equality-of-opportunity-in-machine.html https://research.googleblog.com/2016/10/equality-of-opportun...). It can be trivially extended to discuss hiring. Basically a fair hiring process as defined by the above would be: P(hire | features) = P(hire | race=1, features) = P(hire | race=2, features) This may sound a bit silly or unnecessary, but unless this formula is specifically enforced, if a race were correlated with expected profit of a hire, a hiring algorithm might accidentally infer race from the other features. You can see the paper for an extended discussion. With that background out of the way, let's consider hiring based on quotas. Since hiring takes place continuously across time, in order to maintain a quota across time we would need to change P(hire | race=1) based on the current composition of the company. Therefore P(hire | race=1) != P(hire | race=2) at least some of the time. Therefore it's discriminatory according to the above definition. Of course, you may not like the definition. I would be very interested to hear an alternate formalization of non-discrimination!

justinjlynn 10y ago
Ensuring model safety is indeed a massive pain and is completely non-trivial.
tristanz 10y ago
Unfortunately this isn't enough. Discriminatory rules such as "don't lend to any zip code that's over 75% black" will pass this test if you include this variable as a feature. The same logic applies to less obvious cases such as purchasing behavior.
- torinmr 10y ago
  A particularly hairy case is where you use variables as features which are themselves generated through other potentially-discriminatory processes. For example, including a "has committed a felony" feature seems like a no-brainer for a hiring or lending application, but now any racial or other discrimination present in the criminal justice system has now "infected" your hiring process, so that the outcomes of your hiring process are now racially biased even though your process itself was not.
  - nickpeterson 10y ago
    But at some level, doesn't that just mean reality is biased, and since we don't live in utopia, trying to model one in the small is a sandcastle?
    - inimino 10y ago
      What do you mean by "reality"? Society is pervasively biased, yes, that's the point. That makes it difficult to fight against that bias, but it is necessary if you want to live in a more equal world.
      - dlss 10y ago
        > What do you mean by "reality"? The thing scientists (especially physicists) are trying to model.
        inimino 10y ago
        You're not the person I was asking, and I didn't ask for a definition. In case you missed my point, we aren't talking about physics here, but about society, which is a reality we create. Appealing to "reality" in a discussion about bias amounts to throwing up your hands and dismissing the problem as just "the way things are". It's like if you described a complicated social problem you've observed and I shrugged and said "physics, eh?".
        dlss 10y ago
        > Appealing to "reality" in a discussion about bias amounts to throwing up your hands and dismissing the problem as just "the way things are". Yes, that was how I read GGP's comment. I think you are overstating the degree of control that we have over society. For example, I have often heard the case that society influences young women into roles that eventually prevent them from becoming engineers. Gendered children's toys are often used as an example of this. However, gendered toy preference exists before socialization has occurred (it has been demonstrated in 3-8 month old infants[1]). The same gendered toy preference that exists in humans has also been demonstrated in vervet monkeys[2] and rhesus monkeys[3]. [1] http://link.springer.com/article/10.1007/s10508-008-9430-1 http://link.springer.com/article/10.1007/s10508-008-9430-1 [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643016/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643016/ [3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2583786/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2583786/ With those references now provided, I think it's safe to say gendered toy preference in humans is much bigger than simply a question of which toys we encourage children to buy -- there seems to be a considerable amount of genetics involved. Hopefully now you are understanding what I meant when I defined reality for you. As I said I believe this is also what GGP was referring to. So with that background out of the way: > In case you missed my point, we aren't talking about physics here, but about society, which is a reality we create. > It's like if you described a complicated social problem you've observed and I shrugged and said "physics, eh?". A. How do you know it's a problem with the "reality we create" and not a problem with the reality we are stuck with? (Again, this is how I read GGP) B. If we can trace the problem to something akin to male vervet monkeys preferring to play with Tonka trucks... what are we to do about it beyond ensuring that P(loan | race=1) = P(loan | race=2)? I ask B because the idea of forcing my life choices onto someone else makes me feel ill. It reminds me of being forced to join the basketball team in high school, which I hated (though others seemed to love). The sick feeling compounds when I consider doing it purely on the basis of their race or gender in contexts where their race or gender is causal... and that's usually the course of action people on your side of this discussion recommend.
- dlss 10y ago
  While I agree that `P(loan | race=1, black_zip) = P(loan | race=2, black_zip) = 0` does indeed satisfy the condition, I think the kind of discrimination it creates is in some sense out of scope. This is because if P(loan | black_zip) is profitable > 0, any bank primarily motivated by profit will approve such loans. [this is the statement I believe you meant] If P(loan | black_zip) isn't profitable > 0 after correcting for race, this would mean is that the neighborhood itself signaled something about the person's likelihood of paying back the loan. Perhaps theft is very common, employment is very sparse or seasonal, vandalism/arson of property is common, etc... because we already corrected for race it amounts to saying "don't approve housing loans for neighborhoods where people regularly burn down houses" or similar. This doesn't seem discriminatory to me. So while I do agree there are discriminatory issues not addressed by the formalism, your example seems to be handled by using the normal economic models for what decisions a bank should make given a model P(loan).
  - mjevans 10y ago
    Of course if all banks do that, how is society to deal with this pit/failed neighborhood?
    - dlss 10y ago
      Through products and services not offered by banks?

by dlss 10y ago

justinjlynn 10y ago

Ensuring model safety is indeed a massive pain and is completely non-trivial.

tristanz 10y ago

Unfortunately this isn't enough. Discriminatory rules such as "don't lend to any zip code that's over 75% black" will pass this test if you include this variable as a feature. The same logic applies to less obvious cases such as purchasing behavior.

torinmr 10y ago
A particularly hairy case is where you use variables as features which are themselves generated through other potentially-discriminatory processes. For example, including a "has committed a felony" feature seems like a no-brainer for a hiring or lending application, but now any racial or other discrimination present in the criminal justice system has now "infected" your hiring process, so that the outcomes of your hiring process are now racially biased even though your process itself was not.
- nickpeterson 10y ago
  But at some level, doesn't that just mean reality is biased, and since we don't live in utopia, trying to model one in the small is a sandcastle?
  - inimino 10y ago
    What do you mean by "reality"? Society is pervasively biased, yes, that's the point. That makes it difficult to fight against that bias, but it is necessary if you want to live in a more equal world.
    - dlss 10y ago
      > What do you mean by "reality"? The thing scientists (especially physicists) are trying to model.
      - inimino 10y ago
        You're not the person I was asking, and I didn't ask for a definition. In case you missed my point, we aren't talking about physics here, but about society, which is a reality we create. Appealing to "reality" in a discussion about bias amounts to throwing up your hands and dismissing the problem as just "the way things are". It's like if you described a complicated social problem you've observed and I shrugged and said "physics, eh?".
        dlss 10y ago
        > Appealing to "reality" in a discussion about bias amounts to throwing up your hands and dismissing the problem as just "the way things are". Yes, that was how I read GGP's comment. I think you are overstating the degree of control that we have over society. For example, I have often heard the case that society influences young women into roles that eventually prevent them from becoming engineers. Gendered children's toys are often used as an example of this. However, gendered toy preference exists before socialization has occurred (it has been demonstrated in 3-8 month old infants[1]). The same gendered toy preference that exists in humans has also been demonstrated in vervet monkeys[2] and rhesus monkeys[3]. [1] http://link.springer.com/article/10.1007/s10508-008-9430-1 http://link.springer.com/article/10.1007/s10508-008-9430-1 [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643016/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643016/ [3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2583786/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2583786/ With those references now provided, I think it's safe to say gendered toy preference in humans is much bigger than simply a question of which toys we encourage children to buy -- there seems to be a considerable amount of genetics involved. Hopefully now you are understanding what I meant when I defined reality for you. As I said I believe this is also what GGP was referring to. So with that background out of the way: > In case you missed my point, we aren't talking about physics here, but about society, which is a reality we create. > It's like if you described a complicated social problem you've observed and I shrugged and said "physics, eh?". A. How do you know it's a problem with the "reality we create" and not a problem with the reality we are stuck with? (Again, this is how I read GGP) B. If we can trace the problem to something akin to male vervet monkeys preferring to play with Tonka trucks... what are we to do about it beyond ensuring that P(loan | race=1) = P(loan | race=2)? I ask B because the idea of forcing my life choices onto someone else makes me feel ill. It reminds me of being forced to join the basketball team in high school, which I hated (though others seemed to love). The sick feeling compounds when I consider doing it purely on the basis of their race or gender in contexts where their race or gender is causal... and that's usually the course of action people on your side of this discussion recommend.
dlss 10y ago
While I agree that `P(loan | race=1, black_zip) = P(loan | race=2, black_zip) = 0` does indeed satisfy the condition, I think the kind of discrimination it creates is in some sense out of scope. This is because if P(loan | black_zip) is profitable > 0, any bank primarily motivated by profit will approve such loans. [this is the statement I believe you meant] If P(loan | black_zip) isn't profitable > 0 after correcting for race, this would mean is that the neighborhood itself signaled something about the person's likelihood of paying back the loan. Perhaps theft is very common, employment is very sparse or seasonal, vandalism/arson of property is common, etc... because we already corrected for race it amounts to saying "don't approve housing loans for neighborhoods where people regularly burn down houses" or similar. This doesn't seem discriminatory to me. So while I do agree there are discriminatory issues not addressed by the formalism, your example seems to be handled by using the normal economic models for what decisions a bank should make given a model P(loan).
- mjevans 10y ago
  Of course if all banks do that, how is society to deal with this pit/failed neighborhood?
  - dlss 10y ago
    Through products and services not offered by banks?