5 ms·

If you use data such as repayment history, income, assets, and other metrics highly related to whether or not somebody is going to pay back a loan, and the resu

If you use data such as repayment history, income, assets, and other metrics highly related to whether or not somebody is going to pay back a loan, and the resulting model does not output equal representation among arbitrary population groups, I fail to see a problem. If the data being used to train the model has absolutely no knowledge of arbitrary group identifiers, such as "gender" or "race," the resulting output quite simply not biased. However, if something like "race" or "gender" is actually being used as a feature input, then the output from most ML strategies is highly likely to pick up on correlations between racial and gender groups and certain outputs. That is undoubtedly going to lead to negative discriminatory outputs. So while I see absolutely no problem with the first scenario I mentioned, and I clearly see a problem with the second scenario...I'm inclined to believe that the politics of many people would lead them to have a problem with both.

skybrian 8y ago
The thing that makes this trickier than it looks is that these features aren't necessarily encoded in a straightforward way. For example, race could be inferred with some loss of accuracy from zip code. Add more variables and it might be inferred with more accuracy. Machine learning doesn't necessarily do what you want it to do; it probably will "cheat" and use an easier proxy variable, unless you watch out for it. There is the urban legend about the machine learning program that learned to "recognize" tanks based on time of day in the training set. It probably never happened [1] but it gets the point across. [1] https://www.gwern.net/Tanks https://www.gwern.net/Tanks
- ralusek 8y ago
  Ya, I agree with you that it's definitely tricky business. Including features like zipcode in something like a loan payment predictor is a good example of a sloppy proxy for the underlying and relevant features. Especially when certain data becomes hard to resist using, when something like zipcode of residence might have a 90% overlap with income range and be extremely trivial to acquire, whereas getting detailed and accurate income information from an applicant might be much harder. Tricky stuff.
pooya13 8y ago
I am not saying that this is the case in your examples, but sometimes these "predictors" create an unfair feedback loop. e.g. Cops patrol majority black neighborhoods more because of higher misdemeanor rate and because misdemeanor is a strong predictor of crime. This higher patrol rate leads to higher arrests (for crimes such as having weed on you that might not have a significant difference among races and neighborhoods). The arrest data is then fed back into the system, leading to even more patrols, essentially creating a forced equilibrium. Whereas if we didn't use the model the system might have naturally evolved to a different state. This is one possibility of why these models might be effective in the short term but not "fair" (i.e. not effective in the long term). But in any case, I think it is obvious that the benefits of a private company (such as the insurance company) are not necessarily aligned with the benefits of the society.
- ralusek 8y ago
  Fair points.
- lambdaphagy 8y ago
  Misdemeanors are not a predictor of crime, they are crime. Further, many types of misdemeanors are quality of life issues that have outsize impacts on the people who live in those neighborhoods. Go to a town hall meeting in a bad neighborhood and see what people complain about. It's often not so much burglaries as the guys who are chronically drinking and breaking bottles on the corner at 2 am. It's not terribly enlightened to criticize the police for going after QOL violations you would never tolerate in your own neighborhood. As for whether police attention creates feedback loops, we can check by looking at crime victimization surveys, which generally show the same patterns of racial disparities as arrest records. For that matter, the black / white murder gap has been stable at about 6-8:1 for as long as we've been keeping track, and you can't change that by selectively hassling certain people for weed.
  - pooya13 8y ago
    I fail to see how crime victimization surveys disprove possibility of feedback loops. You can have higher murder rates among the black population, yet have overall similar misdemeanour rates. You can even have a higher misdemeanour rate among the black population but have a propostionally even higher arrest rates due to these models and their feedback loops. Or maybe I am missing something in your argument? Care to elaborate?
denzil_correa 8y ago
> If you use data such as repayment history, income, assets, and other metrics highly related to whether or not somebody is going to pay back a loan, and the resulting model does not output equal representation among arbitrary population groups, I fail to see a problem. If the data being used to train the model has absolutely no knowledge of arbitrary group identifiers, such as "gender" or "race," the resulting output quite simply not biased. Discrimination isn’t solely measured by input but also by outputs. https://en.wikipedia.org/wiki/Disparate_impact https://en.wikipedia.org/wiki/Disparate_impact

skybrian 8y ago

The thing that makes this trickier than it looks is that these features aren't necessarily encoded in a straightforward way. For example, race could be inferred with some loss of accuracy from zip code. Add more variables and it might be inferred with more accuracy. Machine learning doesn't necessarily do what you want it to do; it probably will "cheat" and use an easier proxy variable, unless you watch out for it. There is the urban legend about the machine learning program that learned to "recognize" tanks based on time of day in the training set. It probably never happened [1] but it gets the point across. [1] https://www.gwern.net/Tanks https://www.gwern.net/Tanks

ralusek 8y ago
Ya, I agree with you that it's definitely tricky business. Including features like zipcode in something like a loan payment predictor is a good example of a sloppy proxy for the underlying and relevant features. Especially when certain data becomes hard to resist using, when something like zipcode of residence might have a 90% overlap with income range and be extremely trivial to acquire, whereas getting detailed and accurate income information from an applicant might be much harder. Tricky stuff.

pooya13 8y ago

I am not saying that this is the case in your examples, but sometimes these "predictors" create an unfair feedback loop. e.g. Cops patrol majority black neighborhoods more because of higher misdemeanor rate and because misdemeanor is a strong predictor of crime. This higher patrol rate leads to higher arrests (for crimes such as having weed on you that might not have a significant difference among races and neighborhoods). The arrest data is then fed back into the system, leading to even more patrols, essentially creating a forced equilibrium. Whereas if we didn't use the model the system might have naturally evolved to a different state. This is one possibility of why these models might be effective in the short term but not "fair" (i.e. not effective in the long term). But in any case, I think it is obvious that the benefits of a private company (such as the insurance company) are not necessarily aligned with the benefits of the society.

ralusek 8y ago
Fair points.
lambdaphagy 8y ago
Misdemeanors are not a predictor of crime, they are crime. Further, many types of misdemeanors are quality of life issues that have outsize impacts on the people who live in those neighborhoods. Go to a town hall meeting in a bad neighborhood and see what people complain about. It's often not so much burglaries as the guys who are chronically drinking and breaking bottles on the corner at 2 am. It's not terribly enlightened to criticize the police for going after QOL violations you would never tolerate in your own neighborhood. As for whether police attention creates feedback loops, we can check by looking at crime victimization surveys, which generally show the same patterns of racial disparities as arrest records. For that matter, the black / white murder gap has been stable at about 6-8:1 for as long as we've been keeping track, and you can't change that by selectively hassling certain people for weed.
- pooya13 8y ago
  I fail to see how crime victimization surveys disprove possibility of feedback loops. You can have higher murder rates among the black population, yet have overall similar misdemeanour rates. You can even have a higher misdemeanour rate among the black population but have a propostionally even higher arrest rates due to these models and their feedback loops. Or maybe I am missing something in your argument? Care to elaborate?

denzil_correa 8y ago

> If you use data such as repayment history, income, assets, and other metrics highly related to whether or not somebody is going to pay back a loan, and the resulting model does not output equal representation among arbitrary population groups, I fail to see a problem. If the data being used to train the model has absolutely no knowledge of arbitrary group identifiers, such as "gender" or "race," the resulting output quite simply not biased. Discrimination isn’t solely measured by input but also by outputs. https://en.wikipedia.org/wiki/Disparate_impact https://en.wikipedia.org/wiki/Disparate_impact